Cloudflare to Block Mixed AI Web Crawlers by Default from September 15
Cloudflare will start blocking mixed AI web crawlers that combine search, training, and agent functionalities on all pages by default, requiring separate solutions for each type.
Starting September 15, Cloudflare will block all mixed AI web crawlers on its infrastructure by default. These crawlers use a single type of bot for search, model training, and AI agent tasks. The company now demands that providers separate their solutions into distinct categories: search, training, and agent.
Cloudflare explained that publishers and website owners want to maintain visibility in search and AI services but are unwilling to provide content for free. The new policy aims to give them more control over how their content is accessed by automated systems.
Source: ForkLog