Direct AI crawlers with managed robots.txt

Protect your website or application from AI crawlers by implementing a robots.txt file on your domain to direct AI bot operators on what content they can and cannot scrape for AI model training.

Cloudflare's managed robots.txt explicitly disallows known bots engaged in scraping for AI purposes and AI agent activity.

AI bots are expected to follow the robots.txt directives. Otherwise, they risk getting banned.

To implement a robots.txt file on your domain based on your plan:

Bot Fight Mode

1. Log in to the Cloudflare dashboard ↗ and select your account and domain.
2. Go to Security > Bots.
3. Select Configure Bot Fight Mode.
4. On Manage bot traffic with robots.txt, select the toggle.

Super Bot Fight Mode

1. Log in to the Cloudflare dashboard ↗ and select your account and domain.
2. Go to Security > Bots.
3. Select Configure Super Bot Fight Mode.
4. On Manage bot traffic with robots.txt, select the toggle.

Bot Management for Enterprise

1. Log in to the Cloudflare dashboard ↗ and select your account and domain.
2. Go to Security > Bots.
3. Select Configure Bot Management.
4. On Manage bot traffic with robots.txt, select the toggle.

Note

When you enable the robots.txt file on your domain, existing robots.txt files will be replaced.

Availability

Managed robots.txt for AI crawlers is available on all plans.