Key takeaways
- On 30 October 2026 Google announced that it will support a shared robots.txt token,
__usage_ai-training, in 2027, based on the work of the IETF’s AIPREF group. For now it is a plan. - At Google, that token without its own Google-Extended line counts as Google-Extended blocked, and Google-Extended also covers your site being used as a source in Gemini Apps.
- Bing will honour two separate preferences, training and grounding, from 1 February 2027, and stops reading NoArchive and NoCache as an AI preference on 1 March 2027.
- If you want AI assistants to cite you, the prudent course is to leave AI training open at Google and review NoArchive and NoCache before March 2027.
What changes if you want to opt out of AI training?
A shared way of asking is on its way, but it does not work yet. On 30 October 2026 Google announced that in 2027 it will read a robots.txt token that lets a site ask not to be used for AI training. Bing will honour separate training and citation preferences from 1 February 2027. At Google, opting out with that token can also remove you from Gemini.
Both announcements came out on the same day, on the Google Search Central blog and on the Bing blog. Google describes a “policy token” based on AIPREF, the IETF group working on preferences for AI use of content. It places it “in 2027”, alongside other providers it does not name.
Its crawler documentation, checked on 2 November 2026, goes a step further. The token will be called __usage_ai-training and Google will support it “in early 2027”. It is a plan with a rough date, and it should be treated as one.
OpenAI is less certain. Search Engine Journal reported on 1 November 2026 that OpenAI will implement it “in the coming months”. OpenAI’s own page would not load for us, so we do not treat that as confirmed. On Anthropic, all we can confirm is that the author of the technical draft works there.
What the new robots.txt line looks like
The syntax comes from a draft that Kevin Kelley, of Anthropic, published at the IETF on 30 October 2026. It is titled “Machine-Readable Training Preference Signal” and is an individual draft that the IETF itself marks as not endorsed. It expires on 3 May 2027 unless it is renewed. Its literal example is these three lines.
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/Those lines ask for no page on the site to be used for training, except the ones in the /press-kit/ folder. The draft adds no new directives to the robots.txt standard, RFC 9309. It reuses User-Agent, Allow and Disallow, and matches the token regardless of case.
Two details will slip past most of the templates circulating online. According to the draft, a group with User-Agent: * expresses no training preference at all, so the token has to be named. And the token does not control crawling. Whether a crawler visits your site is still decided by the line for its own user agent.
There is also a second proposal on the table. The AIPREF working group has its own document, written by Gary Illyes (Google) and Martin Thomson (Mozilla). Its rule is different and can sit in robots.txt or in an HTTP header, in the form Content-Usage: train-ai=n. Google names Kelley’s token in its documentation, but as of 2 November 2026 neither syntax is a standard.
Is AI training the same as being cited in AI answers?
No. They are two different uses of your content, and each one is decided separately. Training means using your pages to modify the learned parameters of a model, in the definition used by the AIPREF vocabulary. Grounding means the assistant reads your page at the moment it answers and cites it as a source.
Classic search, the list of results on Google or Bing, is a third thing. Each one is governed by its own mechanism, and the October changes only touch the first two.
| Use | What it is | How Google controls it | How Bing controls it |
|---|---|---|---|
| Training | Your content adjusts a future model | Google-Extended; in 2027, the shared token too | Preference in robots.txt or at page level, from 1 February 2027 |
| Grounding | The assistant reads your page as it answers and cites you | In Gemini Apps and Vertex AI, Google-Extended. AI Overviews and AI Mode follow Search | Page-level preference only, independent of training |
| Classic search | Appearing in the list of results | Googlebot. Google-Extended has no effect on it | Bingbot. No change |
For almost every business we work with, the decision that matters is the second one. Whether a future model has read your site counts for little next to Gemini or Copilot naming you when a buyer asks about your service. We cover it in more depth in our guide to SEO for AI search engines, which includes the user agents for each assistant.
Why can opting out of AI training at Google take you out of Gemini?
Because at Google both decisions hang on the same line. Google does not say so in a single sentence. It is our reading of two official pages, checked on 2 November 2026, and we present it as such.
The crawler documentation explains that Google-Extended decides whether content Google crawls may be used to train future Gemini models. It also decides whether that content can ground answers in Gemini Apps and in the “Grounding with Google Search” feature of Vertex AI. The 30 October blog post adds the other piece. If a site opts out with the shared token and says nothing about Google-Extended, Google will treat it as if Google-Extended were blocked.
Taken together, the two pages lead to a practical consequence. Anyone who copies only the token lines with Disallow: / will most likely stop being a source in Gemini Apps. And anyone who adds an Allow for Google-Extended, which the blog says keeps them in Gemini Apps, is also allowing Gemini training. As things stand, we see no way at Google to say no to training and yes to being cited in Gemini.
What does not change, according to the same documentation, is Search. Google says Google-Extended does not affect a site’s inclusion in Google Search and is not a ranking signal, and the announcement does not mention AI Overviews or AI Mode. Those results depend on Googlebot crawling, as we explain in our guide to Google’s AI Mode. If Google changes its documentation before 2027, this reading will change with it.
At Bing, NoArchive and NoCache stop counting for AI on 1 March 2027
Bing brings the only change with dates that force a decision. Its blog post of 30 October 2026 separates two independent preferences for the first time. The training preference can be set in robots.txt, with the same __usage_ai-training token its examples use, or at page level, while the grounding preference is page level only. Classic search stays as it is, because ranking in Bing results does not count as grounding.
| Date | What happens at Bing |
|---|---|
| 15 December 2026 | A temporary grounding setting opens in Bing Webmaster Tools, only for sites that have NoArchive or NoCache on that date |
| 1 February 2027 | Bing starts honouring the new training and grounding preferences |
| 1 March 2027 | NoArchive and NoCache are no longer read as a training or grounding preference. The temporary setting must be registered before this date |
| 30 September 2027 | Latest end date for the temporary setting, which also ends earlier if Bing recrawls the page |
The change affects anyone who uses noarchive or nocache today to keep Copilot away from their content. That protection switches off on 1 March 2027. Until then there are two routes, registering the temporary setting in Bing Webmaster Tools from 15 December or moving to the new page-level preference. According to Bing’s post, it is written <meta name="Bingbot" content="ai-grounding:disallow">, or ai-training:disallow for training, and it also works as an X-Robots-Tag header.
If your site carries neither tag, there is nothing urgent to do at Bing. From February you will be able to refuse training without giving up being cited in its answers.
What to do, depending on the type of business
The answer depends on what matters more to your business, being cited or keeping your content out of training. This table sums up what we recommend as of 2 November 2026, based on what Google and Bing have published.
| Profile | What it wants | What we recommend |
|---|---|---|
| B2B company, online shop or service business | To be named by AI as an option | Keep Google-Extended allowed and do not copy the shared token before deciding |
| Publisher or content creator | To keep its content out of training | Block Google-Extended or the token, accepting the exit from Gemini Apps, and use the training preference at Bing |
| Business with one section it is happy to share | To share only the press room or the documentation | A general Disallow plus an Allow for that folder, as in the draft’s example, once the token is supported |
For a publisher the maths is different, because the content is the product. Google is already testing payments to publishers for content used in AI answers, and that route may be worth more than a block. Whatever your profile, review what you already have before changing anything.
- Open your robots.txt and find the lines for Google-Extended, GPTBot, ClaudeBot and any
User-agent: *group withDisallow: /. - Search for
noarchiveandnocachein the robots meta tags of your pages and in theX-Robots-Tagheader your server sends. - Check that your CDN or firewall is not rejecting AI crawlers as if they were malicious traffic.
- Write down what you want for training and for citation, provider by provider, with the date of the decision.
- Review it again in February 2027, when Bing starts applying its preferences and Google confirms how it reads the token.
In audits we have found sites that were shutting these crawlers out without realising it, nearly always because of a CDN rule. That is why we check it in every technical SEO review. Our site analyser shows in a couple of minutes which AI crawlers yours is blocking.
If your goal is to be cited by AI, we recommend leaving training open at Google and not copying the new line until Google publishes how it applies it. Anyone using NoArchive or NoCache should decide before 1 March 2027. Blocking is the easy part to get wrong. The hard part is being chosen as a source. That is the job of SEO for AI, and it starts with a review of what you have open and what you do not.
Frequently asked questions
If I block Google-Extended, will I disappear from Google or from AI Overviews?
According to Google’s documentation, checked on 2 November 2026, Google-Extended does not affect inclusion in Search and is not a ranking signal. What you do lose is your site being used as a source in Gemini Apps and for grounding in Vertex AI.
Is User-agent: * with Disallow: / enough to stop AI training on my site?
Kevin Kelley’s draft says that group expresses no training preference: the token has to be named. On top of that, a blanket Disallow shuts the site to every crawler that respects it, Googlebot and Bingbot included, and in practice takes you out of search.
Does blocking now remove content already used for training?
Do not count on it. A robots.txt preference is read on the crawler’s future visits. Neither Google nor Bing says in its 30 October 2026 announcement that it will remove content from models that have already been trained.
Does robots.txt force crawlers to comply?
No. It is a request, and the IETF draft contains no mechanism to enforce it. Google and Bing have announced that they will read these preferences, but a bot can ignore them. In that case, the answer is to block it at the server, the CDN or the firewall.
Can I allow training on just one part of my site?
Yes, going by the draft’s own example. A general Disallow for the token and an Allow for the folder you want to share, such as your press room, will do it. It takes effect once each provider supports the token, which Google places in 2027.
