Starting September 15, 2024, network infrastructure provider Cloudflare will introduce automatic blocking of all "mixed" AI-based web crawlers. This decision will affect pages served through the company's network and marks a new milestone in the battle for content control in the era of generative AI.
"Mixed" refers to bots that are simultaneously used for search indexing, training language models, and operating AI agents. Cloudflare requires developers to clearly separate these functions: search, training, and agent crawlers must be isolated from each other. If the same bot performs all three tasks, it will be blocked.
The reason for this tightening is growing discontent among publishers and website owners. They want to maintain visibility in search engines and AI services but are not willing to give away their content for free to train models and generate instant answers. Cloudflare notes that the internet risks turning into a "zero-click" ecosystem, where chatbots collect original material while completely bypassing original sources. We are witnessing a shift from SEO to AEO (Answer Engine Optimization), and now to GEO (Generative Engine Optimization).
In addition to copyright protection, the new measure aims to reduce infrastructure load. According to Cloudflare, more than 50% of traffic from AI crawlers involves repeated scanning of pages, which rarely provides real value. This creates excessive server load, especially for small websites.
The changes will affect all new Cloudflare clients, existing users, as well as sites on the free plan. Hosting owners will be able to manually disable the blocking if they deem it necessary.
In parallel, the company is developing a commercial model for publishers — Pay Per Use. This involves payment not only for access to content but also for instances where the material proves valuable to an AI service. In the first phase, Cloudflare is collaborating with Ceramic.ai and You.com. Publishers receive payments if their materials appear in chatbot search results.
As an analyst, I see this step as a logical continuation of the trend: internet infrastructure is ceasing to be a neutral platform and is beginning to actively filter traffic. Cloudflare, controlling a significant share of global web traffic, is effectively becoming a regulator in this area. The question is how effective such measures will be against constantly evolving AI bots, and whether they will lead to the fragmentation of the network that we are already observing.