Cloudflare updates bot controls to block AI training and agent crawlers on ad-supported sites by September
Cloudflare is introducing new bot control settings to automatically block AI training and agent crawlers on ad-supported websites starting September 15.
1. New Cloudflare Bot Controls
Cloudflare has introduced a new classification system for web crawlers, replacing its previous single-switch block for AI bots. The company has categorized traffic into three types: Search, which indexes pages for future reference; Agent, which covers real-time automated systems like ChatGPTâs fetch bot; and Training, which pulls content to build model weights. These controls were made available to all customers, including those on the free tier, starting July 1, 2026.
2. Default Blocking Policy
Beginning September 15, 2026, Cloudflare will implement new default settings for all newly onboarded domains, new sites created by existing customers, and all existing free-tier accounts. Under these defaults, crawlers classified as Agent or Training will be blocked on pages that display advertisements, while Search crawlers will remain permitted. Cloudflareâs stated rationale is that ad-supported content is intended for human consumption, and bots that extract this information without providing a referral do not align with the site owner's interests. Users have the option to adjust these settings manually before the September deadline.
3. Impact on AI and Publishers
The policy change creates significant implications for AI developers, as many agentic tools rely on open access to ad-supported web content for tasks like price monitoring and research. Because Cloudflare operates at the network level, these blocks are more effective than traditional robots.txt suggestions. The update also presents a challenge for mixed-use crawlers, such as Googlebot, which performs both search and training functions; blocking Training may inadvertently restrict search visibility. Cloudflare CEO Matthew Prince noted that the company hopes these measures will encourage developers to distinguish between search and training activities. Meanwhile, publishers are encouraged to review their tier status and consider the trade-offs between blocking training data and maintaining search engine traffic.