AI Readiness Checker
Check whether AI crawlers can reach a site, whether it publishes llms.txt and structured data, and how well its pages are built to be quoted as answers.
AI assistants and answer engines find and cite web content through crawlers, the same way search engines do. If those crawlers are blocked, or if your pages bury their answers under layout and boilerplate, your content is less likely to be retrieved and quoted. This checker looks at the technical and structural signals you control.
Enter a domain and the check opens on app.search.co, where search.co's crawler fetches the live site, reads its robots.txt, looks for an llms.txt file and structured data, and scores how answerable the content appears. The initial check needs no account at all; a free search.co account is required to run further checks.
This is a hygiene check, not a ranking prediction. Nobody outside the AI companies can see exactly how their systems select sources, so the aim here is to remove obvious barriers and to separate the settings that matter from the ones that are still speculative.
What the AI readiness check covers.
The report groups its findings into four areas, each answering a different question about how AI systems can interact with the site.
- AI crawler access: how your robots.txt rules apply to known AI user-agents such as
GPTBot,ClaudeBot,PerplexityBotand theGoogle-Extendedtoken, so you can see which are allowed and which are blocked. - llms.txt: whether the site publishes an
/llms.txtfile and, if so, whether it can be fetched. - Structured data: whether pages carry schema markup, which gives machines an explicit description of what a page is about.
- Answerability score: search.co's own assessment of how clearly the content states answers that could be extracted and quoted.
Each AI user-agent controls something different.
The most common mistake in this area is treating every AI token as the same switch. They are not. Vendors separate their crawlers by purpose, and blocking one may leave others untouched. Read the access section of the report against the purposes below, as documented by each vendor at the time of writing, and check the vendors' own pages for changes.
| Token | Operated by | What it is used for |
|---|---|---|
GPTBot | OpenAI | Collecting content that may be used to train models |
OAI-SearchBot | OpenAI | Surfacing sites in ChatGPT's search features, separate from training |
ClaudeBot | Anthropic | Collecting content for model training; Anthropic documents separate agents for search and user-requested fetches |
PerplexityBot | Perplexity | Indexing pages so they can appear and be cited in Perplexity answers |
Google-Extended | Whether Google may use crawled content for Gemini models; it does not affect Google Search inclusion or ranking |
Two consequences follow. First, blocking a training crawler and blocking a search or retrieval crawler are separate decisions with separate trade-offs, and many publishers choose to block one but not the other. Second, Google-Extended is not how you opt out of AI features in Google Search. Those features rely on normal Googlebot crawling, so the controls are the standard Search ones, such as nosnippet, not this token.
Remember that robots.txt is a request that well-behaved crawlers honor, not an access control. If content must not be fetched, it needs authentication. The robots.txt tester is useful for confirming exactly how a rule matches a given user-agent and path.
Where llms.txt and schema actually stand.
llms.txt is a proposed convention, introduced in 2024, for a Markdown file at the site root that summarizes a site and points language models to its most useful pages. It is easy to publish and does no harm. It is also important to be clear about its status: no major search engine or AI assistant has confirmed that it uses llms.txt as a ranking, retrieval or citation signal. The report tells you whether the file exists; treat a missing one as a low-priority, low-effort item, not a problem.
Structured data is on firmer ground. Schema markup is an established, documented way to describe entities, products, articles, organizations and FAQs to machines, and it powers rich results in search. Whether specific AI systems weight it is not publicly documented, but accurate markup removes ambiguity about what a page is, which is useful regardless. Validate it against what is visible on the page; markup that describes content users cannot see violates Google's structured data guidelines.
Making the access decision, then fixing answerability.
Decide on crawler policy at the business level before touching the file. Publishers whose content is the product may reasonably block training crawlers while allowing search and retrieval agents, since retrieval can send visits and citations. A B2B company that wants to be recommended by assistants generally has little reason to block anything. Write the decision down, implement it per token, and re-run the check to confirm the rules behave as intended.
Then work on answerability, which tends to matter more than any single file. Pages that are easy to quote state the answer near the top in plain sentences, use descriptive headings that match the questions people ask, define terms explicitly and keep key facts in text rather than images or scripts that need rendering. A page that opens with three paragraphs of preamble before answering is harder for any system, human or machine, to use.
For deeper page-level structure, pair this with the SEO score checker, which reviews headings, titles and on-page basics on the homepage.
Mistakes in AI readiness work.
- Blanket blocking by accident: a
Disallow: /underUser-agent: *blocks AI crawlers along with everyone else, and is a common leftover from staging sites. - Blocking Google-Extended to escape AI Overviews: that token does not govern Search features, so it will not have that effect.
- Treating llms.txt as a ranking lever: it is unconfirmed as a signal; do not prioritize it over content structure.
- Shipping schema that does not match the page: markup should describe visible content accurately, or it creates risk instead of clarity.
- Forgetting CDN and firewall rules: bot protection can block AI crawlers even when robots.txt allows them, so check server logs if access looks wrong.
Running an AI readiness review.
Run the check
Enter the domain and review crawler access, llms.txt, schema and the answerability score.
Agree a crawler policy
Decide per token whether training, search and retrieval agents should be allowed.
Update robots.txt
Implement rules per user-agent, then re-run the check to confirm the result.
Improve key pages
Rewrite top pages so answers appear early, in plain text, under question-shaped headings, with accurate schema.
AI readiness questions.
Does blocking GPTBot remove my site from ChatGPT?
Does Google-Extended affect my Google rankings?
Do I need an llms.txt file?
What does the answerability score measure?
Is robots.txt enough to stop AI companies using my content?
Will schema markup get me cited by AI assistants?
Pair it with these checks.
Want your content structured for AI answers?
SEO.co's AI search team audits crawler access, schema and content structure, and rewrites priority pages so they are easier for search engines and assistants to quote.