Cloudflare Docs
Visit Cloudflare Fundamentals on GitHub
Set theme to dark (⇧+D)

When does Cloudflare crawl my site?

Cloudflare may crawl or make HTTP requests to your site to make sure its protected and performing properly.

Crawling situations

Specific products

Cloudflare will crawl your site when you have specific products enabled:

  • Always Online
    • User-Agent: Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +
  • SSL/TLS recommender
    • User-Agent: Cloudflare-SSLDetector
    • This crawler ignores your robots.txt file unless there are rules explicitly targeting the user agent.
  • Load balancing monitors
    • User-Agent: Mozilla/5.0 (compatible; Cloudflare-Traffic-Manager/1.0; +; pool-id: <POOLID>)
  • Health checks
    • User-Agent: Mozilla/5.0 (compatible; Cloudflare-Traffic-Manager/1.0; +; healthcheck-id: <HEALTHCHECK_ID>)

Other situations

Cloudflare will also crawl your site in other, specific situations: