01
Overview
Cloudflare’s new AI crawler defaults take effect on September 15, 2026, changing how new domains—and some existing free customers that did not choose different settings—handle bots used for AI search, agents and model training.
The important change is not simply that Cloudflare can block AI bots. It is that Cloudflare now separates AI-related crawling into three behaviors: Search, Agent and Training. Site owners can treat those behaviors differently, which creates a more precise choice between discoverability and reuse.
02
What changed on September 15, 2026
Cloudflare says new domains onboarding to its network now default to allowing Search bots while blocking Training and Agent bots on pages that display ads. Its July announcement also says the changes apply to existing free customers that had not changed their settings before September 15. Customers can change these policies in the dashboard.
Cloudflare’s documentation defines the categories this way:
- Search crawlers collect or index content so it can later be used to answer searches or questions.
- Agent traffic acts in real time on behalf of a person, including chat-fetch bots and browser-use agents.
- Training crawlers collect content to train or fine-tune models.
Each behavior can be allowed, blocked across a zone, or blocked only on pages where Cloudflare detects ads, depending on the available policy configuration.
03
The mixed-purpose crawler change matters for SEO
The most consequential detail is how Cloudflare handles crawlers with more than one purpose.
Cloudflare says multi-purpose crawlers that combine Search and Training are now evaluated according to all applicable behaviors. Because the most restrictive applicable rule wins, Cloudflare specifically warns that Googlebot, Applebot and BingBot can be blocked for customers that choose to block Training under configurations affected by this policy.
That does not mean blocking AI training automatically destroys Google rankings. Cloudflare does not make that claim, and search-ranking impact depends on which crawler is blocked, what behavior classification applies, and how a search provider uses that crawler.
It does mean site owners should no longer assume that a generic “block AI training” choice is isolated from search discovery when the same crawler performs multiple functions.
04
Cloudflare Search vs Agent vs Training
The three-category model is useful because these activities represent different exchanges for a publisher or business.
Search generally exists to make content discoverable later and can send people back to the source. Agent traffic may retrieve or interact with a site in real time for a user. Training uses content to improve a model and may not produce a direct visit at all.
A site can therefore want Search access while rejecting Training access. Cloudflare’s newer controls are designed around that distinction, although mixed-purpose crawlers complicate it when an operator does not separate its behaviors cleanly.
05
What happened to “Block AI bots”?
Cloudflare’s current bot documentation marks the legacy Block AI bots setting as deprecating on September 15, 2026. The newer configuration uses behavior-based AI bot policies instead.
This is worth checking even if a site previously configured the old toggle. A legacy rule may not express the same intent as separate Search, Agent and Training choices.
06
How to review your Cloudflare AI crawler policy
Cloudflare’s current documentation points users to Security Settings and the AI bot policy configuration. Review Search, Agent and Training separately rather than treating every automated crawler as one category.
For a site that depends on organic discovery, the safest conceptual order is:
- Decide whether search discovery should remain available.
- Decide whether real-time agents should be able to fetch or act on the site.
- Decide whether model-training crawlers may use the content.
- Check how Cloudflare classifies mixed-purpose crawlers relevant to the site.
- Recheck robots.txt and edge-level behavior after changing a policy.
Do not copy another publisher’s settings blindly. An ad-supported publication, ecommerce site, documentation site and private application can have very different incentives.
07
Bot Preference Sync and robots.txt
Cloudflare has also announced Bot Preference Sync, which reflects AI bot configuration in robots.txt. Cloudflare says it can prepend generated preferences while preserving existing Disallow directives.
There is an important distinction between preference and enforcement. robots.txt communicates instructions to cooperating crawlers; Cloudflare’s edge controls can enforce blocks against traffic it identifies. Site owners with complex custom crawler rules may prefer to manage those policies manually.
Because Bot Preference Sync has had its own rollout path, verify its current availability in your Cloudflare account before assuming that a dashboard policy has rewritten robots.txt.
08
Does this affect Google indexing?
Potentially, but not in one universal way.
Cloudflare explicitly names Googlebot among mixed-purpose crawlers that can be affected when Training is blocked under the new policy logic. That makes crawler configuration an SEO-sensitive setting.
However, Cloudflare does not document a guaranteed ranking or indexing outcome from any particular policy. Search engines also operate multiple crawlers and product surfaces. The correct response is to verify actual crawl access and Search Console behavior rather than infer a ranking penalty from the setting alone.
09
Why this matters beyond Cloudflare
The change reflects a broader shift in web infrastructure: “AI crawler” is no longer a useful single category.
Search indexing, model training and user-directed agents have different purposes. Cloudflare is turning those distinctions into infrastructure-level controls, which means SEO and content-distribution decisions increasingly overlap with AI data policy.
For publishers, marketers and site owners, the practical lesson is simple: decide which forms of machine access create value for you, then configure them deliberately.
10
FAQ
When did Cloudflare’s new AI crawler defaults take effect?
September 15, 2026.
What are Cloudflare’s three AI crawler categories?
Search, Agent and Training.
Are Training and Agent bots blocked everywhere by default?
No. Cloudflare’s stated new-domain default is to block Training and Agent on pages that display ads while allowing Search. Existing-account behavior can depend on plan and prior configuration.
Can blocking AI training affect Googlebot?
Cloudflare says mixed-purpose crawlers including Googlebot can be blocked when Training is blocked under applicable configurations. That is why site owners should review the exact policy rather than assume training and search are always separable.
Is robots.txt enough to block an AI crawler?
robots.txt is a preference that cooperating crawlers can honor. Edge enforcement is a separate mechanism.
Should every website allow Search and block Training?
Not necessarily. The right policy depends on how the site values discoverability, agent access, content licensing, privacy and machine reuse.
Sources
Primary and supporting sources
Use the linked documentation to check current pricing, access and product behaviour. The dates above describe this article; they do not guarantee that every vendor claim remains current.