Cloudflare AI Controls - decide which AI bots read your site, and charge themFor any founder whose product IS the content (docs, media, a SaaS with a public knowledge base), the only choice used to be "block all AI or allow all". Cloudflare now splits AI traffic by intent, so you keep search while cutting off free training scrapes.
- Three bot classes - Search (indexing, allowed), Agent (ChatGPT/Claude acting live), Training (model data) - each toggled separately
- "Content Use Signals" in robots.txt - tag content immediate / reference / full to control how bots store and reshare it
- From Sept 15, 2026, new domains block Training + Agent on ad pages by default; multi-purpose crawlers like Googlebot follow the strictest matching rule
- The pay-per-crawl marketplace lets you charge crawlers instead of only blocking them
Strongest when ad or subscription revenue rides on your own content - the intent split finally lets you keep Google while pricing out free training scrapes; niche if your site is a thin app shell with nothing worth crawling.
https://blog.cloudflare.com/content-independence-day-ai-options/📎
Read also:→
The non-branded query that silences founders→
Firecrawl /monitor - alerts when docs change→
HasData - SERP parsing eats 20-50% of dev time