Skip to content
Cloudflare Docs
DirectoryAPISDKsChangelog

Cloudflare crawlers

Cloudflare may crawl or make HTTP requests to your site to make sure its protected and performing properly.

Crawling situations

Specific products

Cloudflare will crawl your site when you have specific products enabled:

  • Always Online
    • User-Agent: Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online)
  • Health checks
    • User-Agent: Mozilla/5.0 (compatible; Cloudflare-Healthchecks/1.0; +https://www.cloudflare.com/; healthcheck-id: <HEALTHCHECK_ID>)
    • HEALTHCHECK_ID is a 16-character string associated with the health check ID.
  • Load balancing monitors
    • User-Agent: Mozilla/5.0 (compatible; Cloudflare-Traffic-Manager/1.0; +https://www.cloudflare.com/traffic-manager/; pool-id: <POOL_ID>)
    • POOL_ID is a 16-character string associated with the load balancing pool ID being monitored.
  • Prefetch URLs
    • User-Agent: Mozilla/5.0 (compatible; CloudFlare-Prefetch/0.1; +http://www.cloudflare.com/)
  • SSL/TLS recommender
    • User-Agent: Cloudflare-SSLDetector
    • This crawler ignores your robots.txt file unless there are rules explicitly targeting the user agent.
  • Security Insights
    • User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 (compatible; +https://developers.cloudflare.com/security-center/)

Other situations

Cloudflare will also crawl your site in other, specific situations:

  • Speed tests
    • User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36 PTST/190628.140653
    • Triggered when: You launch a speed test from within the Cloudflare dashboard.
  • Support diagnostics:
    • User-Agent: Cloudflare-diagnostics
    • Triggered when: Cloudflare Support Engineers perform error checks and by continuous monitoring used to raise intelligent alerts in the Cloudflare dashboard.
  • Custom Hostname validation:
    • User-Agent: Cloudflare Custom Hostname Verification
    • Triggered when: You choose to validate a custom hostname with an HTTP ownership token.

Developer product crawlers

Browser Run is a Cloudflare developer product that can crawl third-party websites on behalf of Cloudflare customers. Unlike the crawlers in the Crawling situations section, it does not crawl your site as part of a Cloudflare service you have enabled. It is used by developers building applications with Cloudflare's platform.

The Browser Run /crawl endpoint uses a User-Agent of CloudflareBrowserRenderingCrawler/1.0. For non-configurable headers, bot detection IDs, and cryptographic verification methods you can use to identify or block Browser Run traffic, refer to Automatic request headers.

AI Search is a Cloudflare developer product that indexes your website content so it can be searched. The AI Search crawler only crawls a website you own (the domain must exist in the same Cloudflare account) that you select as your data source in AI Search.

The crawler uses a User-Agent of Cloudflare-AI-Search. You can allow or block it with WAF rules using its bot detection ID.