Changelog
All customers can now manage AI crawlers by behavior — Search, Agent, and Training — instead of a single Block AI bots toggle. Configure these options from Block AI Bots. New defaults, in which Training and Agent are blocked on pages that display ads while Search remains allowed, take effect for new domains on September 15, 2026.
Enterprise Bot Management customers can now use BotBase, a searchable directory of all tracked bots and agents with their behavior classification and detection IDs, and Attribution Business Insights, a dashboard showing site-wide and per-operator crawl-to-referral ratios alongside bot traffic to your content.
Cloudflare will prepend our managed robots.txt before your existing robots.txt, combining both into a single response.
Web Bot Auth is an authentication method that leverages cryptographic signatures in HTTP messages to verify that a request comes from an automated bot. This provides a more robust way of verifying bots.
Events detected by the anomaly detection engine are now given a bot score of 2.
Machine Learning model v9 is now the default model for all new zones and existing zones set to use the latest machine learning model.
Direct AI crawlers on what they can and cannot scrape from your website or application by implementing a robots.txt file to your domain.
You can now create a Bot Detection Alert to notify you when Cloudflare detects a spike in Bot traffic on your website.
AI bots protection has been upgraded from a custom rule to a managed rule.