How to tell if your website is ready for AI search: three checks and their sources

How do you know if your website is ready for AI search?

Your website is ready for AI search if it passes three checks. The search crawlers of ChatGPT, Claude and Perplexity can get in. Your pages meet Google’s two requirements for AI Overviews and AI Mode. And your robots.txt says what you want about training, a decision each company documents separately.

Passing them means nothing technical is keeping you out. Whether an AI assistant cites you also depends on what you publish and on who talks about you, and no checklist can promise that. What these checks do is rule out the failures that drop you from the candidate list without anyone noticing.

The table brings the five reviews together, with where to look and which signal should worry you.

Can the search crawlers of ChatGPT, Claude and Perplexity get in?

They can if none of three layers stops them: robots.txt, the CDN or firewall, and the server response. Each company runs one crawler for its search engine and another that fetches pages when a user asks. The first decides whether you are shown in their search, and the second whether they can read your page when someone asks for it. The training crawlers are a separate matter, covered further down.

Which agent decides what

OpenAI’s bot documentation separates three agents. OAI-SearchBot surfaces websites in ChatGPT search, and sites that opt out do not appear in those answers. GPTBot crawls to train the models. ChatGPT-User visits pages when a user asks for something, and OpenAI warns that robots.txt rules may not apply to it.

Anthropic and Perplexity follow the same pattern, according to Anthropic’s help article on its crawlers, dated 7 April 2026, and Perplexity’s bot guide.

OpenAI adds two practical details. Each bot is configured separately, and a robots.txt change can take around 24 hours to show up in its search. Anthropic asks you to repeat any block on every subdomain, which matters for European businesses that run de., fr. or en. subdomains: each one needs its own robots.txt check.

The most-specific-group rule

The most common mistake lies in how groups are read. RFC 9309 is the robots.txt standard, published by the IETF in September 2022. Under its text, a crawler obeys the * group only if no other group names it. Google’s robots.txt specification works the same way, with the most specific group taking precedence over the rest.

User-agent: *
Disallow: /private/

User-agent: OAI-SearchBot
Allow: /

With this file, OAI-SearchBot can also crawl /private/, because its own group does not repeat the restriction. If you want a named bot to respect a closed folder, its group needs the Disallow line as well. When several bots share the same rules, the standard allows several User-agent lines in a row above them.

The CDN and the hosting layer

A correct robots.txt is not enough if the server does not let the bot through. Google says so on its “AI features and your website” page, which asks that crawling be allowed in robots.txt “and by any CDN or hosting infrastructure”. It is listed among its best practices rather than its technical requirements, but it is Google’s own wording.

Cloudflare has two settings worth checking. Its managed robots.txt, when switched on, adds a Disallow: / for GPTBot, ClaudeBot, Google-Extended and five other agents to your file. Nobody on your team wrote those lines. OAI-SearchBot is not on that list, but blocking a crawler from AI Crawl Control creates a firewall rule that can answer with a 403. Google’s specification adds that a robots.txt file only applies to its own host, protocol and port, so each country domain needs its own check.

The third failure is harder to spot. A hosting anti-bot protection can answer with a JavaScript challenge page and a 200 status code. A crawler that does not run JavaScript gets that page instead of yours, and the logs look perfectly normal.

How to check

1. Open yourdomain.com/robots.txt and look for the OAI-SearchBot, Claude-SearchBot, Claude-User and PerplexityBot groups. If they are missing, the * group applies to them.
2. Check the CDN dashboard for a managed robots.txt or an active AI crawler block.
3. Run the page through our robots.txt and no-JavaScript HTML analyser.
4. Search the server logs for those agents, if you have access, and see which status code they get. Match the IPs against the lists published by OpenAI, Anthropic and Perplexity.

It helps to know what the analyser covers and what it leaves out. It checks whether robots.txt fully blocks seven agents: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and CCBot. It does not cover Claude-SearchBot or Claude-User, does not report the page’s HTTP status code and downloads with its own user agent. So it cannot see what your firewall serves to each bot, and a partial block or a group shared by several agents has to be read by hand.

Does your website meet Google’s two requirements for AI Overviews and AI Mode?

Google asks for two things. The page has to be indexed and eligible to appear in Search with a snippet. And the site has to be included in the “Search generative AI” control in Search Console. The first comes from its page on AI features. The second was added by Google’s guide to generative AI features, which John Mueller announced on 15 May 2026.

Meeting them guarantees nothing, and Google warns that it does not promise indexing or that a page will be shown. What the two requirements do is open the door. Almost everything that shuts it lives on your own site.

Indexed and eligible for a snippet

Open URL Inspection in Search Console with your home page and confirm it is indexed. Then look for noindex, nosnippet and max-snippet in the robots meta tag and in the server’s X-Robots-Tag header. Google documents them as the way to limit what its AI features take from your page, which is why they keep you out if someone left them in by mistake.

The classic case is an inherited template, with a nosnippet added on a staging site that travelled to production. The analyser checks the meta tag noindex, but it does not detect nosnippet, max-snippet or the X-Robots-Tag header. That part is a manual check.

The “Search generative AI” control

It sits in Search Console, under Settings. According to the Search Console help page on that control, it has three options: Include, which is the default, Exclude, and Inherit from the parent property. Since 31 August 2026 it has been available for every website.

Exclude removes your links and content from AI Overviews, AI Mode and the generative AI features in Discover, and takes a few days to apply. Google makes clear that it is not a ranking signal and does not affect the rest of Search. If your country sites are separate Search Console properties, check the setting on each one. One change by a colleague is enough to remove you from those answers, and nothing on the site gives it away.

Proof that you appear

On 3 June 2026 Google announced a generative AI performance report in Search Console, which has reached every website since 31 August. It counts impressions only, in those Search and Discover features, broken down by page, country and date. If a page has impressions there, it meets both requirements.

Does your robots.txt say what you want about training?

Training a model and citing you in an answer are different uses, and robots.txt separates them with different agents. GPTBot and ClaudeBot are the training crawlers of OpenAI and Anthropic. Blocking them does not take you out of ChatGPT or Claude search, which rely on OAI-SearchBot and Claude-SearchBot. The check is whether those lines reflect a decision or a copied template.

At Google the line is blurrier. According to its common crawlers documentation, Google-Extended does not affect inclusion in Search and is not a ranking signal. It also governs grounding in Gemini Apps and the Vertex AI API, which is the use of your site as a source when answering. A site that blocks it to keep its content out of Gemini training also stops being a source in those answers.

Google also plans to read a shared token, __usage_ai-training, in 2027, to ask that a site not be used for training. What changes and what to do depending on your type of business is in our piece on the shared robots.txt line to opt out of AI training.

The text has to be in the HTML, not only in JavaScript

The content that matters should arrive in the HTML the server sends. Google processes JavaScript, but its JavaScript SEO basics guide still recommends server-side rendering or prerendering, because not every bot can run it. OpenAI, Anthropic and Perplexity do not say in their documentation whether their crawlers do.

A study by Vercel with MERJ, published on 17 December 2024, counted 569 million GPTBot requests and 370 million Claude requests in one month. With that sample, taken on the Vercel network, it concluded that none of the major AI crawlers executed JavaScript, with Gemini and AppleBot as the exceptions. They downloaded the files without running them. It is a snapshot from late 2024, and should not be read as a permanent rule.

To check, open the page in your browser and use “View page source”. If the main text is in that source, a crawler that does not run JavaScript receives it too. The analyser counts visible words without running JavaScript, so a low count on a page that looks full on screen means the text is arriving through JavaScript. The fix then lies in web development, with server-side rendering or static pages.

What you do not need to check

Google is explicit about this. Its May 2026 guide says you do not need new machine-readable files, AI text files, markup or Markdown, and that Google Search ignores llms.txt. Nor does it ask you to chunk content, write in a special way or add structured data. From its search engine’s point of view, AEO and GEO are “still SEO”.

FAQs and markup still make sense. They help a page be understood and help readers find the answer, which is why we recommend them. What cannot be claimed is that Google requires them or that you get cited for having them. To decide which markup is worth the effort, our guide to structured data for AI search sets out the priorities.

That list applies to Google. OpenAI, Anthropic and Perplexity publish nothing equivalent on how they choose sources, so outside Google it is wise to distrust any recipe. What you should not do is in writing, and we covered it when we explained what Anthropic treats as manipulating AI sources.

Where to start to get your website ready for AI search

Start with what can be seen from outside. Run your home page and one service page through The Baller Company’s website analyser, which asks for no email and does not store the URL. Fix what it flags in robots.txt and the HTML. Then go into Search Console, confirm indexing and check that the “Search generative AI” control is still set to “Include”. Leave the CDN dashboard for last.

If you would rather we review and fix it for you, content included, that is our SEO for AI work. How assistants choose their sources is covered in our guide to SEO for AI search engines.

Frequently asked questions