Cloudflare Will Block AI Crawlers by Default From September 15, 2026 - What It Means for Data Collection

By Marcus Reiner · 2026-07-26 · 12 min read · News

Cloudflare is closing the open web to AI crawlers by default starting September 15, 2026. GPTBot traffic is up 147% in a year, Meta-ExternalAgent up 843%. Here is what this means if your work depends on public web data.

What Cloudflare announced

On July 1, 2026, Cloudflare announced that starting September 15, 2026, it will block 'mixed-use' AI crawlers by default on any page that carries ads. A mixed-use crawler is one that blends traditional search indexing with AI training or AI agent data collection without giving site owners a way to control which use case applies. Pure search crawling remains allowed by default; training and agent access do not, unless the site owner explicitly changes the setting.

The new defaults apply to new Cloudflare customers, new sites created by existing customers, and - notably - all existing free-tier accounts, unless those site owners opt out before the deadline. Sites already on paid Cloudflare plans keep their current settings unless they change them manually. Cloudflare's CEO Matthew Prince framed the move around the fact that non-human traffic now makes up the majority of all internet traffic, arguing the ecosystem needs enforced, not just polite, controls.

Alongside the blocking change, Cloudflare is replacing its original Pay Per Crawl feature (launched in 2025) with a new model called Pay Per Use. The distinction matters: Pay Per Crawl charged AI companies per page fetch, which created an odd incentive since more than half of AI crawl traffic was reportedly re-fetching pages that had not changed since the last visit. Pay Per Use instead compensates publishers when an AI system actually references their content in a generated answer, not simply when a bot visits the page.

The scale of the shift, in real numbers

GPTBot request volume rose 147% over the past year, and Meta-ExternalAgent traffic rose 843% over the same period, reflecting how rapidly AI crawling has scaled relative to traditional search indexing. Overall AI-related bot traffic climbed over 300% between January 2025 and mid-2026.

More than 2.5 million websites already actively disallow AI training crawlers via robots.txt as of mid-2026, and GPTBot specifically is blocked by roughly 19% of sites that publish a robots.txt file. Since Cloudflare alone routes an estimated one-fifth of all web traffic, its policy change effectively sets a default posture for a very large share of the internet, not just its direct customers.

Search-oriented crawlers - the traditional kind that exchange crawl access for sending human visitors via search results - dropped to just 10.7% of total crawl traffic measured in some analyses, illustrating how thoroughly AI-oriented crawling has come to dominate non-human web traffic.

Why robots.txt alone was never enough

Robots.txt has always been an honor-system control - a request, not an enforcement mechanism. A well-behaved crawler like GPTBot or ClaudeBot checks and respects it; a crawler with no interest in cooperating simply ignores it, and historically there has been no technical barrier stopping that. Cloudflare's July 2026 change moves a meaningful slice of the web from that honor system to an enforced permission system at the network edge, where compliance is no longer optional for crawlers routed through Cloudflare's infrastructure.

This is directly relevant to how ToptierProxy.com itself operates: earlier in 2026 we specifically reviewed and corrected our own robots.txt and Cloudflare AI Crawl Control settings after finding they were blocking legitimate AI assistant traffic we wanted to allow. The lesson generalizes - site owners increasingly need to actively manage both layers (robots.txt and their CDN's crawler controls) rather than assuming either one alone reflects their real intent.

What this means if your work depends on public web data

The blocks specifically target crawlers that announce themselves as AI training or agent bots via User-Agent strings and IP ranges, from datacenter infrastructure. This is a meaningful distinction: the policy is not blocking 'scraping' in a general sense, it is blocking self-identified AI crawlers operating from recognizable datacenter infrastructure that has not been separated by declared function.

For legitimate public-data collection - price monitoring, market research, SEO tracking, brand protection - the practical effect is that identifying yourself honestly as an AI crawler from datacenter IPs is becoming a much less viable path on Cloudflare-protected sites. Collection through a real browser on a residential or mobile IP, behaving as a normal visitor accessing public pages without circumventing explicit access barriers, remains the path that survives this shift, the same pattern already required for DataDome, Amazon and LinkedIn as covered elsewhere on this site.

If your data collection genuinely functions as an AI training pipeline at meaningful scale, Cloudflare's Pay Per Use model is worth engaging with directly rather than routing around - it is a legitimate, sanctioned path to the same data with cooperating publishers, and the September 15 deadline is explicitly Cloudflare's invitation for AI companies to declare their crawler function honestly rather than operating as an undifferentiated mixed-use bot.

What site owners should actually do

If you run a site on Cloudflare, you now control AI crawler access in two layers rather than one: robots.txt for well-behaved bots, and Cloudflare's dashboard-level AI Crawl Control (available even on the Free plan) for actual enforcement against bots that ignore your stated preferences. Decide deliberately rather than accepting whatever default lands on your account come September 15.

A reasonable default for most content sites: block AI training crawlers (they give you nothing back - no referral traffic, no attribution), allow AI search/answer crawlers that drive human visitors your way, and allow user-initiated agent fetches (an AI reading your page on a specific user's behalf functions similarly to a human visit). The exception is content that is your paid product - gated courses, proprietary datasets, metered journalism - where blocking the agent category too makes sense, since an AI reading and relaying your paywalled content to a user functions as a paywall bypass regardless of intent.

Frequently Asked Questions