01
Overview
Cloudflare now lets site owners manage AI-related crawlers by behavior instead of relying on one broad AI-bot switch. The three categories are Search, Agent and Training, and the right configuration depends on whether you want discovery, real-time agent access, model-training access, or some combination of them.
02
Before changing anything
Write down the outcome you actually want. For many public sites, “block AI” is too vague.
You may want search engines and answer engines to discover your pages while preventing model-training crawlers from ingesting them. You may want AI agents to fetch public documentation but not interact with sensitive routes. An ad-supported publisher may have different priorities from a SaaS documentation site.
03
Where Cloudflare puts the controls
Cloudflare’s current documentation directs customers to Security Settings and Configure AI bot policies. The available behavior groups are Search, Agent and Training.
Cloudflare documents three mitigation choices for each behavior: block across the zone, block only on pages that display ads, or allow.
04
Step 1: Review Search
Search covers crawlers that collect or index content so it can answer questions later. If organic and AI-search discovery matter to your site, understand what will be blocked before changing this category.
05
Step 2: Review Agent
Agent covers automated activity acting in real time on a person’s behalf, including chat-fetch and browser-use agents. Decide whether those systems should be able to access the public site and whether ad-page-specific blocking matches your business model.
06
Step 3: Review Training
Training covers crawlers that use content to train or fine-tune models. Cloudflare’s September 15 defaults block this category on ad-displaying pages for new domains while Search remains allowed.
The complication is mixed-purpose crawling. A crawler can perform both Search and Training, so a Training block can affect it under Cloudflare’s most-restrictive-rule logic.
07
Step 4: Check mixed-purpose crawlers
Do not skip this step if search visibility matters.
Cloudflare specifically says Googlebot, Applebot and BingBot are examples of multi-purpose crawlers that can be blocked when a customer chooses to block Training under applicable configurations.
That does not prove a ranking loss. It means the configuration can affect crawler access, which should be tested rather than assumed.
08
Step 5: Verify robots.txt separately
Cloudflare’s Bot Preference Sync is designed to reflect AI bot policy choices in robots.txt. robots.txt is still a declared preference for cooperating crawlers, while edge blocking is enforcement.
If you use custom crawler rules or have special agreements with a bot operator, inspect the resulting robots.txt rather than assuming category-wide sync expresses every exception correctly.
09
Step 6: Test the result
After a policy change:
- Load your public robots.txt and review relevant directives.
- Check Cloudflare bot/security analytics for blocked crawler activity where available.
- Test important public routes as appropriate.
- Monitor Google Search Console crawl and indexing signals if Google discovery matters.
- Revisit the policy if desired Search traffic is being caught by a Training rule.
10
A practical policy framework
For a public content site, ask three separate questions: Do I want this content discoverable? Do I want an agent to retrieve it for a user right now? Do I want it used for model training?
Answering those separately is more useful than asking whether you “allow AI bots.”
11
What not to assume
Do not assume every existing Cloudflare account received the same September 15 defaults. Do not assume blocking Training is always isolated from Search. Do not assume robots.txt itself enforces a block. And do not assume a crawler’s classification will never change.
Cloudflare’s model is useful precisely because these policies are becoming more granular. Treat them as distribution settings, not merely security toggles.
Sources
Primary and supporting sources
Use the linked documentation to check current pricing, access and product behaviour. The dates above describe this article; they do not guarantee that every vendor claim remains current.