Cloudflare introduces a total block on "hybrid" AI crawlers: a new standard for content protection
Starting September 15, network infrastructure provider Cloudflare will automatically block all "mixed" AI-based web crawlers. This decision fundamentally changes the rules of the game for publishers and website owners who have long struggled with uncontrolled data collection.
The block applies to bots used simultaneously for search indexing, model training, and AI agent functionality. Cloudflare requires a clear separation of functions: search crawlers, training systems, and agent solutions must operate independently of each other.
Why This Matters for the Market
Company representatives emphasize that publishers want to maintain visibility in search engines and AI services but are no longer willing to give away their content for free. We are witnessing a fundamental shift: the internet is moving from the SEO era to AEO (Answer Engine Optimization), and then to GEO (Generative Engine Optimization). As noted in Cloudflare's statement, "The internet risks transitioning to a 'zero-click' ecosystem, where chatbots collect original content for instant answers, completely bypassing original sources."
More than 50% of traffic from AI crawlers consists of repeated page scans that rarely provide value. The new measure not only protects content but also reduces the load on web page infrastructure. The change will affect new Cloudflare customers, existing users' websites, and all free-tier clients. Hosting owners can manually disable the block, but this will require a conscious choice.
Commercial Model for Publishers
In parallel, Cloudflare is launching a Pay Per Use system—a commercial scheme that involves payment not only for access to material but also when content is valuable to an AI service. In the initial phase, the company is working with Ceramic.ai and You.com. Publishers receive payments if their materials appear in chatbot search results.
Expert Commentary: This decision is a logical step amid the fragmentation of the internet. Cloudflare is essentially creating a precedent that could become an industry standard. Publishers gain a monetization tool, while AI companies get clear rules of the game. However, questions remain: how will compliance with bot "purity" be verified, and will this lead to new technical barriers for legitimate developers.