Changelog

Subscribe to RSS

2025-08-27

Enhanced crawler insights and custom 402 responses

We improved AI crawler management with detailed analytics and introduced custom HTTP 402 responses for blocked crawlers. AI Audit has been renamed to AI Crawl Control and is now generally available.

Enhanced Crawlers tab:

• View total allowed and blocked requests for each AI crawler
• Trend charts show crawler activity over your selected time range per crawler

Custom block responses (paid plans): You can now return HTTP 402 "Payment Required" responses when blocking AI crawlers, enabling direct communication with crawler operators about licensing terms.

For users on paid plans, when blocking AI crawlers you can configure:

• Response code: Choose between 403 Forbidden or 402 Payment Required
• Response body: Add a custom message with your licensing contact information

Example 402 response:

HTTP 402 Payment Required
Date: Mon, 24 Aug 2025 12:56:49 GMT
Content-type: application/json
Server: cloudflare
Cf-Ray: 967e8da599d0c3fa-EWR
Cf-Team: 2902f6db750000c3fa1e2ef400000001

{
  "message": "Please contact the site owner for access."
}

2025-07-01

Introducing Pay Per Crawl (private beta)

We are introducing a new feature of AI Crawl Control — Pay Per Crawl. Pay Per Crawl enables site owners to require payment from AI crawlers every time the crawlers access their content, thereby fostering a fairer Internet by enabling site owners to control and monetize how their content gets used by AI.

For Site Owners:

• Set pricing and select which crawlers to charge for content access
• Manage payments via Stripe
• Monitor analytics on successful content deliveries

For AI Crawler Owners:

• Use HTTP headers to request and accept pricing
• Receive clear confirmations on charges for accessed content

Learn more in the Pay Per Crawl documentation.

2025-07-01

AI Crawl Control refresh

We redesigned the AI Crawl Control dashboard to provide more intuitive and granular control over AI crawlers.

• From the new AI Crawlers tab: block specific AI crawlers.
• From the new Metrics tab: view AI Crawl Control metrics.

To get started, explore:

• Manage AI crawlers.
• Analyze AI traffic.

2024-09-23

AI Crawl Control

Every site on Cloudflare now has access to AI Audit, which summarizes the crawling behavior of popular and known AI services.

You can use this data to:

• Understand how and how often crawlers access your site (and which content is the most popular).
• Block specific AI bots accessing your site.
• Use Cloudflare to enforce your robots.txt policy via an automatic WAF rule.

To get started, explore AI audit.