Cloudflare AI Crawler Block (September 2026): What Changed and What to Do
By Marcus Reiner · 2026-07-26 · 7 min read read · News
Cloudflare now lets sites block Search, Agent and Training bots separately, with new defaults from 15 September 2026. See what changed, who is affected, and what data collectors should do.
Short answer: what did Cloudflare change?
From 15 September 2026, Cloudflare's default for newly onboarded domains is to block bots it classifies as AI Training or AI Agent on pages that display ads, while Search crawlers stay allowed. Cloudflare announced the change on 1 July 2026, together with new controls that let any customer, including Free plan users, set Search, Agent and Training separately.
It is a default for site owners, not a ban. Owners can block each category on all pages, block it only on pages that display ads, or not block it at all, and they could opt out of the new defaults before 15 September. For anyone who collects web data, the practical result is that more of the web is now set up to refuse AI-labelled traffic unless the owner has said yes.
What Cloudflare actually announced
Cloudflare splits AI traffic into three categories. Search is crawling that builds a database of a site in order to answer later queries. Agent is automated activity on a person's behalf in real time, such as ChatGPT-User or browser-use agents. Training is collecting content to train or fine-tune models.
For each category an owner picks one of three settings: block on all pages, block only on pages that display ads, or do not block. The controls sit in the site's Security settings in the Cloudflare dashboard and are available on every plan.
Cloudflare also says multi-purpose crawlers, which do search and training with the same bot, are judged by the most restrictive rule that applies to what they do. According to the announcement, Verified bots are no longer allowed by default, and bots that reproduce content in full cannot be Verified.
Two further items are proposals and tests, not rules you must follow today: a content-use signal being tested in robots.txt, and a proposed Forwarded header (RFC 7239) that would let an agent say who it acts for.
Who is affected: new domains, existing sites and what is still unclear
Cloudflare's changelog says new domains onboarding to Cloudflare receive the updated defaults from 15 September 2026. It does not say that existing sites are switched automatically, and it tells owners they can opt out of the new defaults before that date.
Some news coverage says all free-plan sites are affected, and other coverage says existing sites are not reset. Those reports conflict, so check your own dashboard rather than relying on a summary, including this one.
The default also applies only to pages that display ads. A page without ads is not covered by the 'block only on pages that display ads' setting.
Will it block Googlebot, Bingbot and Applebot?
Cloudflare says Search crawling stays allowed by default. The complication is bots that do both jobs. Cloudflare's own post names Googlebot, Applebot and BingBot as multi-purpose crawlers that can be affected on sites that block Training, because the most restrictive rule wins.
Cloudflare lets owners opt out so that training crawlers that also crawl for search keep their current settings. If you own a site on Cloudflare, check how these crawlers are treated before you block Training, and watch Google Search Console and Bing Webmaster Tools for crawl errors after any change.
What it means if you collect web data
Most scrapers and proxy users are not labelled AI crawlers, but the line is blurry. A script that browses on behalf of a user, an AI browser agent, or an LLM tool that fetches pages live fits Cloudflare's definition of an Agent. If a site on Cloudflare blocks Agent traffic on pages with ads, those requests can be challenged or refused.
A proxy changes the IP address a request comes from, not what the traffic is. We read Cloudflare's categories as being about what a bot does and whether it identifies itself, so changing IPs would not turn an agent into a search crawler. Cloudflare does not publish how it classifies every bot, so treat that as our reading, not a guarantee.
The bigger signal is the trend. Site owners now have a one-click way to say no to AI use of their content, and more of them will use it. Data collection plans that assume open access to every public page are getting less reliable.
How to collect data responsibly on Cloudflare-protected sites
1. Check the site's robots.txt and terms before you collect anything, and keep a record of what you checked.
2. Look for an official API, data feed or licensed dataset first. It is more stable than scraping and it settles the question of permission.
3. Identify your bot with an honest user agent and contact details, crawl slowly, and cache what you fetch so you do not request the same page twice.
4. Ask for permission when the data matters to your business. Cloudflare launched a Pay Per Crawl marketplace in July 2025 so that AI companies can pay publishers for access, and some publishers will license data directly.
5. For routine monitoring of public pages, such as prices, search results and ad checks, use residential proxies from a provider that sources its IPs with consent, keep request rates low, and use a real browser only when a page needs JavaScript.
6. Do not try to get around a block a site owner has set on purpose. Doing so can breach the site's terms and, depending on where you are, the law.
If you own a Cloudflare site: how to check your settings
Open your site's Security settings in the Cloudflare dashboard and look for the AI traffic options. Check three things: what Search, Agent and Training are each set to; whether the 'only on pages that display ads' option applies to your pages; and whether the Google, Bing and Apple crawlers you rely on for visitors are still being crawled.
If you want AI assistants to be able to find and cite your pages, keep Search allowed. Block Training only if you are comfortable with the trade-off, and test after every change.
What we could not confirm
We could not confirm whether existing sites receive the new defaults automatically, the full list of bots in each category, or the terms of any pay-per-use model. Some articles say pay-per-use replaces Pay Per Crawl, but Cloudflare's announcement does not say that.
An earlier version of this article quoted traffic growth figures and robots.txt blocking percentages. We removed them because we could not trace them to a primary source. This article is based on Cloudflare's own announcement and changelog, and we will update it if Cloudflare publishes more detail.
Frequently Asked Questions
Is Cloudflare blocking all AI crawlers?
No. From 15 September 2026 the new default for newly onboarded domains blocks bots classified as AI Training or AI Agent on pages that display ads. Search crawlers stay allowed, and site owners can change each setting.
When does Cloudflare's AI crawler block start?
15 September 2026 is the date Cloudflare gave for the new defaults. Owners could opt out of the new defaults before that date.
Does the change affect existing Cloudflare sites?
Cloudflare's changelog says new domains receive the updated defaults. It does not say existing sites are changed automatically, and news reports disagree. Check your Security settings in the dashboard.
Will Cloudflare block Googlebot?
Search crawling stays allowed by default. But Cloudflare says multi-purpose crawlers are judged by the most restrictive rule, and it names Googlebot, Applebot and BingBot as examples that can be affected on sites that block Training. Owners can opt out to keep current settings for those crawlers.
Can I still scrape websites that use Cloudflare?
Many Cloudflare sites remain reachable, but owners now have simple controls to refuse AI Training and Agent traffic, and Cloudflare also offers separate bot protection. Check robots.txt and the site's terms, look for an official API or licensed data first, and do not try to get around a deliberate block.
What is Pay Per Crawl?
Pay Per Crawl is a Cloudflare marketplace, launched in July 2025, that lets publishers charge AI companies for access. Cloudflare's July 2026 announcement does not change it, although some articles describe newer pay-per-use models.
How do I allow or block AI crawlers on my Cloudflare site?
Use the AI traffic options in your site's Security settings. You can set Search, Agent and Training separately to block on all pages, block only on pages that display ads, or not block.
Do proxies get around Cloudflare's AI crawler block?
A proxy changes your IP address, not what your traffic is. Cloudflare's categories are about what a bot does and whether it identifies itself. Proxies are not a reliable or appropriate way to override a block a site owner has chosen.