← Free toolsavailableRBT

Robots.txt Tester

Fetch a host's robots.txt, see how it parses into user-agent groups, and test whether a specific path is allowed or blocked for a specific crawler.

Enter the host you want to test, for example www.example.com or shop.example.com. The tester opens on app.search.co, where search.co's crawler fetches /robots.txt from that host, parses the groups and rules, and lets you check a path against a chosen user-agent. The answer comes from the live file, so it reflects what crawlers see today rather than what is in your repo.

There is no signup for your first test. For more, create a free search.co account and keep going.

Robots.txt is short, but its matching rules are easy to misread, and a single character can block a whole directory. The sections below explain how crawlers like Googlebot actually pick a group and resolve conflicting rules, what robots.txt can and cannot do for indexing, and how to handle the growing list of AI crawler user-agents.

// what you get

What the tester shows.

The goal is to answer one precise question, "can this bot fetch this URL?", and to show why.

Fetch result: whether robots.txt was reachable and what status it returned, since a missing file and a failing server are treated very differently.
Parsed groups: each User-agent group with its Allow and Disallow rules, plus any Sitemap lines.
Path test: an allowed or blocked verdict for the path and user-agent you choose.
Matching rule: the specific rule that decided the verdict, which is what you need to fix an unexpected block.

// how it works

Host scope, group selection and longest match.

Scope is per host. A robots.txt file applies only to the exact protocol, host and port it is served from. The file at https://www.example.com/robots.txt says nothing about https://example.com/, https://blog.example.com/ or http://www.example.com/. Subdomains, CDNs serving assets from another host, and staging environments each need their own file, which is why the tester takes a host rather than a root domain.

One group applies per crawler. A crawler looks for the group whose User-agent line most specifically matches its own token and follows only that group. If Googlebot finds a User-agent: Googlebot group, it ignores the User-agent: * group entirely. This trips up many sites: someone adds a Googlebot group with one rule and unknowingly removes every rule that applied from the wildcard group.

Within a group, the longest match wins. Google compares the path against every rule and applies the most specific one, measured by the length of the rule path. If an Allow and a Disallow rule are equally long, Google uses the less restrictive, Allow. Order in the file does not matter for Google, though some older parsers used first match, so avoid relying on ambiguity.

Two wildcards are supported. * matches any sequence of characters and $ anchors the end of the URL. Paths are case-sensitive, so /Admin/ and /admin/ are different rules.

Example rules and how a path resolves (hypothetical file)
Rules in the group	Path tested	Result and why
Disallow: /shop/ and Allow: /shop/sale/	/shop/sale/shoes	Allowed: the Allow rule is longer
Disallow: /*?sort=	/shop/?sort=price	Blocked: * matches /shop/
Disallow: /*.pdf$	/files/guide.pdf?v=2	Allowed: $ requires the URL to end in .pdf
Disallow: /private	/private-events/	Blocked: rules are prefixes, so /private matches /private-events/
Disallow: /page and Allow: /page	/page	Allowed: equal length, Google picks the less restrictive

// limits

Robots.txt controls crawling, not indexing.

A disallowed URL can still be indexed. If other pages link to it, Google may list the bare URL in results, usually without a snippet, because it knows the URL exists but cannot read it. Blocking is therefore the wrong tool for keeping a page out of search.

To remove a page from the index, use a noindex meta tag or X-Robots-Tag header, and leave the URL crawlable so the crawler can see that directive. Blocking a URL in robots.txt and adding noindex at the same time is self-defeating: the crawler never fetches the page, so it never reads the noindex. For sensitive content, use authentication, not robots.txt, which is public and readable by anyone.

The same logic applies to your sitemap. If the sitemap validator shows URLs that are also disallowed here, one of the two signals is wrong and should be corrected.

// edge cases

When robots.txt is missing or failing.

How a crawler reacts depends on the status code of the robots.txt request itself. Per Google's documentation, a 4xx response (other than 429) is treated as if no robots.txt exists, so everything is crawlable. A 5xx response or an unreachable server is treated more cautiously: Google may pause crawling the site for a period, fall back to a previously cached copy, and adjust its behavior if the errors persist. The exact timings are Google's to change, so check its current robots.txt documentation rather than relying on fixed numbers.

The practical lesson: a robots.txt that intermittently returns 500, often because a CDN or WAF rule misfires, can slow crawling of a healthy site. Google also stops reading a robots.txt file past a size limit documented at 500 kibibytes, so generated files with thousands of lines risk silently losing rules at the end. Google ignores Crawl-delay; Bing and some other crawlers honor it.

// ai crawlers

Testing AI crawler user-agents.

AI companies publish their own user-agent tokens, and robots.txt is the common way to grant or deny them. Commonly seen tokens include GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, CCBot (Common Crawl), Google-Extended and Applebot-Extended. Note that Google-Extended is a control token, not a separate crawler: it governs whether Google may use content for certain AI models and does not change regular Googlebot crawling or Search.

Decide per bot whether you want retrieval and citation in AI answers, training use, both or neither, then test each token here to confirm the file does what you intended. Because group selection is exclusive, a dedicated User-agent: GPTBot group overrides your wildcard rules for that bot. To see AI access alongside llms.txt and schema signals, run the AI readiness checker.

// common mistakes

Robots.txt mistakes worth testing for.

Run these checks after every deploy that touches the file, and after any migration.

Shipping the staging file: a leftover Disallow: / under User-agent: * blocks the entire site.
Blocking CSS and JavaScript: Google renders pages, and blocked resources can make it misread layout and content.
Using robots.txt to deindex: it stops the crawler from ever seeing the noindex you added.
Forgetting the prefix rule: Disallow: /blog also blocks /blog-news/ and /blogroll. Add a trailing slash when you mean the directory.
Testing the wrong host: rules on www do nothing for the apex domain or a subdomain.

// faq

Robots.txt tester questions.

Does blocking a page in robots.txt remove it from Google?

No. It stops crawling, not indexing. A blocked URL can still appear in results if it is linked from elsewhere. Use a noindex tag on a crawlable page to keep it out of the index.

Which rule wins when Allow and Disallow conflict?

Google applies the most specific rule, meaning the one with the longest matching path. If two rules are the same length, it uses the less restrictive Allow rule.

Does one robots.txt cover all my subdomains?

No. Each protocol, host and port combination has its own robots.txt. A blog subdomain or a separate asset host needs its own file.

What happens if robots.txt returns a server error?

Google treats server errors cautiously and may slow or pause crawling, using a cached copy where it has one. A 404 for robots.txt, by contrast, is read as no restrictions at all.

How do I block AI training crawlers but stay in search?

Add groups for the specific AI user-agent tokens you want to restrict, such as GPTBot or CCBot, and leave Googlebot and Bingbot allowed. Google-Extended controls certain AI uses of your content without affecting Google Search.

Do I need an account to use the tester?

Not for the first test. After that, sign in with a free search.co account to run more.

// related tools