What Cloudflare's allow search, disallow AI training setting puts in your robots.txt

The problem being solved is the mixed-use crawler - a single crawler doing both Search and Training. Previously Cloudflare's Block and Block on pages with ads settings did not apply to those crawlers, because blocking them could also affect search discoverability. That exemption is gone. Block and Block on pages with ads now apply to all training crawlers including mixed-use ones, so selecting Block stops Applebot, Bingbot and Googlebot from reaching your site, search included.

Cloudflare classifies bots by behavior, and a single bot can exhibit more than one. There are three behavior controls, applied at the domain level: Search (crawling to build a search index), Training (crawling to train or fine-tune a model), and Agent (user-directed agents visiting a page on behalf of a human, such as chat fetch bots and browser-use agents). Disallow AI Training is only available as a setting for Training - not for Search and not for Agent.

There is no Disallow AI Training on pages with ads. Cloudflare can detect which pages serve ads, but that list is too large and changes too frequently to enumerate in robots.txt, so an ads-only preference cannot be expressed in that file. Two other things are being retired alongside this: the Block AI Bots setting is deprecated in favor of the granular Search, Training and Agent controls, and Managed Robots.txt is deprecated in favor of Bot Preference Sync, with customers who enabled it migrated to the new system.

Cloudflare created the Accountable designation after talking to crawler operators directly since July. To qualify, an operator must meet or commit to meeting four requirements: a mechanism for site owners to opt out of AI training through robots.txt or a similar standard; a mechanism to opt out of AI summaries set with the operator directly, and next year through Cloudflare; URL-level visibility into which pages were made available for training, along with metrics showing how content appeared in search; and assurance that opting out of AI training will not affect traditional search results.

Apple, Google and Microsoft all demonstrate that they meet the qualifications, and Cloudflare says each honors or has committed, in a specified time frame, to honor the new setting - each combining capabilities available today with time-bound commitments for those still in development. Amazon, Anthropic, Meta and OpenAI are categorized as Accountable for the relevant crawlers on different grounds: they separate their Search and Training crawlers, so Cloudflare can block the Training crawler without affecting search. Cloudflare Radar publicly tracks the controls, transparency and reporting provided by Accountable crawler operators.

The per-operator opt-outs are not uniform, and that decides whether Disallow AI Training does anything for a given crawler. Applebot honors a Disallow rule in robots.txt for Applebot-Extended, accepts AI Summaries preferences via the nosnippet directive in page HTML, and lets content be labeled as paywalled to exclude it from generative output. Googlebot honors a Disallow rule for Google-Extended, offers a toggle inside its webmaster portal to exclude a site's content from generative search results, and provides site owners with metrics and reporting on search results and AI summary results. Apple and Google have both stated that disallowing training does not impact search ranking.

Bing is the gap. Microsoft is currently building the mechanism to respect a no-training preference in robots.txt at the domain/site level, targeted for early 2027. Until that support launches, selecting Disallow AI Training will not automatically convey a no-training preference to Bing through robots.txt - the same practical behavior as the previous Training Block setting, which did not apply to mixed-use crawlers such as Bingbot. Today the Bing path is the NOARCHIVE meta tag, plus Bing's Block URLs or Content Removal tool, and Microsoft has stated that using NOARCHIVE will not impact search ranking. Apple does not yet provide a tool for URL-level inspection and has shared an in-progress solution for next year, while Google's additional URL-level transparency tooling for Google-Extended is expected in the weeks to come.

The unit of enforcement is behavior, not identity, and the migration tables mean training-classified crawlers meet the new preference without site owners doing anything. Domains that had the legacy Block AI Bots setting move to Search Allow, Training Disallow AI Training, Agent Block on pages with ads; the legacy Block on pages with ads setting maps to the same three. Domains that previously configured the granular controls have any Training selection of Block or Block on pages with ads migrated to Disallow AI Training. New domains onboarding from September 15 get one of two presets depending on whether the site earns money from advertising: non-ad sites get Allow across Search, Training and Agent, while ad-supported sites get Allow for Search, Disallow AI Training for Training, and Block on pages with ads for Agent, on Cloudflare's reasoning that ad revenue depends on a human actually seeing the page.

Cloudflare's three behaviors do not name retrieval-augmented generation. The closest defined match for a retrieval-time fetch is Agent - user-directed agents visiting a page on behalf of a human, such as chat fetch bots and browser-use agents - and Cloudflare has deliberately not shipped a Disallow setting for Agents, because the Internet does not yet have a well-established directive for expressing Disallow preferences to agents and it will revisit as standards such as ai-prefs mature. Agents are still blocked outright on ad-serving pages under the ad-supported preset, on the stated reasoning that agents fetch the page with nobody there to see the ads. Where a crawl feeds a model rather than a user, the classification that matters is Training, and separating the search crawler from the training crawler, as Amazon, Anthropic, Meta and OpenAI do, is what makes selective blocking possible at all.

The behavior of agents in the wild is the backdrop. 404 Media reported on September 15, 2026 that it receives a steady stream of emails purporting to be from AI agents, including one that learned it was not blocked by the site's robots.txt page and then emailed an offer to run a $399 audit on which AI crawlers the site blocks.

Commenters on Hacker News were largely unconvinced by the enforcement model. One asked directly what the setting does beyond publishing a preference and whether it blocks IP ranges too; another called Accountable a fancy word for a pinky promise with a label, arguing nothing stops fetched data ending up in a training run afterwards. The sharpest distrust was aimed at the separated-crawler claim: one commenter said they find it difficult to trust that Meta or OpenAI use their separate search and training crawlers only for the respective purposes.

Cloudflare's answer to the general version of this is that the network layer does what the file cannot: publish the preference, identify who is crawling, classify why they are crawling, block the ones that ignore it, and report what each operator actually does on Radar. It makes no claim about preventing misuse of content that has already been fetched.

On the identification side, one commenter said they have been experimenting with a Web Bot Auth implementation as an Envoy dynamic module, and another said they are using it for header-level signed attestations; that is a practitioner direction, not something Cloudflare's announcement describes. Separate complaints concerned Cloudflare classifying any client without a popular JavaScript-enabled browser as a bot, and one commenter's claim that Cloudflare lets what it considers good bots through all of your bot blocking rules with no option to turn this off unless you pay.

Three dates are worth tracking. Bing's robots.txt no-training support is targeted for early 2027, and until it lands Disallow AI Training conveys nothing to Bingbot through robots.txt. Apple's URL-level inspection tool is described as in progress for next year. Cloudflare's stated goal is to let site owners control how much of their content is included in an AI summary - set once on Cloudflare rather than with each operator separately - by early next year; today the summary controls named in the post are per-operator, via nosnippet for Applebot or the webmaster portal toggle for Google.

The announcement does not describe any header or cryptographic signature requirement for crawler operators, and it does not publish the exact robots.txt text Bot Preference Sync emits. It does not address enforcement against crawlers that neither identify themselves nor honor the published preference, beyond Cloudflare's own classification and blocking. Cloudflare frames the remainder as standards work with bodies including the Internet Engineering Task Force, and names ai-prefs as the standard that would eventually give agents a Disallow vocabulary.