Starting September 15, network infrastructure provider Cloudflare is taking web resource protection to a new level: blocking all "mixed" AI-based crawlers will be activated by default. This decision is a direct response to the growing load on publishers' servers and the escalation of intellectual property issues in the era of generative AI.
This refers to bots that combine the functions of search indexing, model training, and AI agent operation in one package. Cloudflare insists on a clear separation of these tasks. Publishers want to maintain visibility in search results and AI services, but are categorically unwilling to give away their content for free to train competing models or generate answers that bypass original sources.
As the company emphasizes, the internet is moving towards a "zero-click" ecosystem, where chatbots consume original content for instant answers, completely ignoring authors. We have already moved from SEO to AEO (Answer Engine Optimization), and now stand on the threshold of GEO (Generative Engine Optimization). This blocking is an attempt to return control to website owners.
The key factor that pushed Cloudflare to tighten measures is statistics. More than 50% of traffic from AI crawlers consists of repeated scans of the same pages, which brings neither benefit nor new data. This is pure infrastructure load without any return.
The new rules will affect all free tier customers, new users, and existing websites. Hosting owners, however, will be able to manually disable this option if they believe that "mixed" bots are beneficial to them.
In parallel, Cloudflare is developing a commercial Pay Per Use model for publishers. Now, payment is charged not just for access to the material, but also in cases where the content is deemed valuable for an AI service. At launch, partners include Ceramic.ai and You.com. Publishers receive payments if their materials appear in chatbot outputs.
Expert opinion: This is a logical step in the fight for content sovereignty. However, the problem is that major players like OpenAI and Google have already accumulated huge volumes of data before such restrictions were introduced. Blocking "mixed" crawlers is more about protecting future content than correcting past mistakes. The market is clearly moving towards a model where every query to AI will be monetized, and Cloudflare is trying to stake its claim as the main arbiter in this new economy.