Cloudflare uses generative AI to fight unauthorized AI crawlers

Subscribe for free access to :arrow_forward: Data Points!

Cloudflare launched AI Labyrinth, a new defense system that generates fake web pages to waste the resources of unauthorized AI crawlers that ignore “no crawl” directives. The system creates convincing but irrelevant content networks that serve as honeypots, helping Cloudflare identify and track unauthorized scrapers. AI crawlers now generate over 50 billion requests daily on Cloudflare’s network, representing nearly 1 percent of all web traffic they handle. This approach marks a shift from traditional blocking methods and could make it difficult for AI crawlers to extract useful data. (Cloudflare)