--- description: Manage a robots.txt file to direct AI bot operators on content scraping permissions. title: robots.txt setting image: https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/og.png?v=4d922f618f245c06 --- [Skip to content](#main-content) > Documentation Index > Fetch the complete documentation index at: https://developers.cloudflare.com/bots/llms.txt > Use this file to discover all available pages before exploring further. # robots.txt setting Last updated Oct 8, 2026|Copy as Markdown| [View as Markdown](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/index.md)| [Agent setup](https://developers.cloudflare.com/agent-setup/) AI companies use crawlers to collect website content for training language models, generating search answers, and other purposes. A `robots.txt` file at the root of your domain tells these crawlers which content they should or should not access. Bot Preference Sync generates managed `robots.txt` directives from your Search, Agent, and Training policies when at least one policy is set to **Block**, **Block on pages with ads**, or, for Training, **Disallow**. `robots.txt` compliance is voluntary. The file expresses your preferences, but it does not prevent crawlers from accessing your content at a technical level. Some crawler operators may disregard your `robots.txt` directives (instructions like `Disallow: /`) and crawl your content regardless. Note To enforce access restrictions, use [AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers/) to block selected AI crawlers. You can use `robots.txt` to express your preferences and AI Crawl Control to enforce crawler-specific blocks. ## Compatibility with existing `robots.txt` files Cloudflare detects whether your origin server already has a `robots.txt` file and adjusts accordingly — either merging with your existing file or creating one from scratch. ### Existing robots.txt file If your website already has a `robots.txt` file — verified by an HTTP `200` response — Cloudflare will prepend our managed `robots.txt` before your existing `robots.txt`, combining both into a single response. For example, without this feature enabled, the `robots.txt` content of `crawlstop.com` would be: *Feature not enabledtxt* ```txt User-agent: * Disallow: /lp Disallow: /feedback Disallow: /langtest Sitemap: https://www.crawlstop.com/sitemap.xml ``` With the managed `robots.txt` enabled, Cloudflare will prepend our managed content before your original content, resulting in what you can view at [https://www.crawlstop.com/robots.txt ↗︎](https://www.crawlstop.com/robots.txt). *Feature enabledtxt* ```txt # As a condition of accessing this website, you agree to abide by the # following content signals: # (a) If a content-signal = yes, you may collect content for the # corresponding use. # (b) If a content-signal = no, you may not collect content for the # corresponding use. # (c) If the website operator does not include a content signal for a # corresponding use, the website operator neither grants nor restricts # permission via content signal with respect to the corresponding use. # The content signals and their meanings are: # search: building a search index and providing search results (e.g., returning # hyperlinks and short excerpts from your website's contents). Search # does not include providing AI-generated search summaries. # ai-input: inputting content into one or more AI models (e.g., retrieval # augmented generation, grounding, or other real-time taking of # content for generative AI search answers). # ai-train: training or fine-tuning AI models. # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET. # BEGIN Cloudflare Managed content User-Agent: * Content-signal: search=yes, ai-train=no, use=reference Allow: / User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: meta-externalagent Disallow: / # END Cloudflare Managed Content User-agent: * Disallow: /lp Disallow: /feedback Disallow: /langtest Sitemap: https://www.crawlstop.com/sitemap.xml ``` ### No robots.txt file If your website does not have a `robots.txt` file, Cloudflare creates a new file with managed `Disallow` rules for known AI crawlers and serves it for you. ## Implementation To add managed directives for AI training crawlers: 1. In the Cloudflare dashboard, go to the **Security Settings** page. [Go to **Settings** ↗](https://dash.cloudflare.com/?to=/:account/:zone/security/settings) 2. Filter by **Bot traffic**. 3. Select **Configure AI bot policies**. 4. For **Training**, select *Disallow*. 5. Turn on **Enable Bot Preference Sync**. 6. Select **Save**. With Bot Preference Sync enabled, setting **Training** to *Disallow* adds managed directives and enables edge blocking for AI training crawlers. Mixed-use crawlers are excluded from this edge block. The **Search** and **Agent** policies also add their respective managed directives when set to *Block* or *Block on pages with ads*. ## Content Signals Policy Content Signals are a set of machine-readable directives in a `robots.txt` file that categorize how crawlers may use your content. The three categories are `search` (building a search index), `ai-input` (feeding content into AI models for real-time answers), and `ai-train` (training or fine-tuning AI models). Domains on the Free plan that do not have their own `robots.txt` file and do not use the managed `robots.txt` feature will display the Content Signals Policy when a crawler requests the `robots.txt` file for your domain. The Content Signals Policy defines these categories but does not express any specific preferences about your content. To set preferences (for example, `ai-train=no`), configure your AI bot policies and enable Bot Preference Sync. *Content Signals Policytxt* ```txt # As a condition of accessing this website, you agree to abide by the # following content signals: # (a) If a content-signal = yes, you may collect content for the # corresponding use. # (b) If a content-signal = no, you may not collect content for the # corresponding use. # (c) If the website operator does not include a content signal for a # corresponding use, the website operator neither grants nor restricts # permission via content signal with respect to the corresponding use. # The content signals and their meanings are: # search: building a search index and providing search results (e.g., returning # hyperlinks and short excerpts from your website's contents). Search # does not include providing AI-generated search summaries. # ai-input: inputting content into one or more AI models (e.g., retrieval # augmented generation, grounding, or other real-time taking of # content for generative AI search answers). # ai-train: training or fine-tuning AI models. # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET. ``` Cloudflare includes the Content Signals Policy whenever Bot Preference Sync adds managed directives to your `robots.txt` file. Caution Google Search Console may occasionally report `Syntax not understood` for Content Signals and newer directives in the `robots.txt` standard. However, we have observed no impact on crawling rates or SEO as a result of these reports. ## Content use signal Cloudflare is testing `content-use`, an optional extension to [Content Signals ↗︎](https://contentsignals.org/) that lives in your `robots.txt`. It adds a fourth field alongside the existing `search`, `ai-input`, and `ai-train` signals to describe what a crawler may keep and reuse after accessing your content. The field takes one of three values, from least to most permissive: | Value | Meaning | | --- | --- | | `use=immediate` | Interact, but store and reuse nothing. | | `use=reference` | Index, excerpt, and link back. | | `use=full` | Summarize and reproduce. | For customers who have turned on the managed `robots.txt` setting, Cloudflare adds `use=reference` to the managed content, in line with the existing default of `search=yes,ai-train=no`: *Managed robots.txt with the content-use signaltxt* ```txt User-Agent: * Content-signal: search=yes, ai-train=no, use=reference Allow: / ``` ## Availability Managed `robots.txt` for AI crawlers is available on all plans. Was this helpful? YesNo ## On this page [![](https://developers.cloudflare.com/_astro/logo.te5VL_aD.svg)Docs](https://developers.cloudflare.com/) ```json {"@context":"https://schema.org","@type":"TechArticle","@id":"https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/#page","headline":"robots.txt setting","description":"Manage a robots.txt file to direct AI bot operators on content scraping permissions.","url":"https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/","inLanguage":"en","image":"https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/og.png?v=4d922f618f245c06","dateModified":"2026-10-08","publisher":{"@type":"Organization","name":"Cloudflare","description":"One platform for your apps, agents, and workforce. Build, secure, and scale without managing infrastructure","url":"https://www.cloudflare.com/","sameAs":["https://github.com/cloudflare","https://www.linkedin.com/company/cloudflare","https://x.com/cloudflare"],"logo":{"@type":"ImageObject","url":"https://developers.cloudflare.com/logo.svg"},"address":{"@type":"PostalAddress","streetAddress":"101 Townsend St","addressLocality":"San Francisco","addressRegion":"CA","postalCode":"94107","addressCountry":"US"},"contactPoint":[{"@type":"ContactPoint","contactType":"Customer Support","url":"https://support.cloudflare.com/","availableLanguage":["English"]},{"@type":"ContactPoint","contactType":"Sales","url":"https://www.cloudflare.com/contact/","availableLanguage":["English"]}]},"isPartOf":{"@type":"WebSite","@id":"https://developers.cloudflare.com/#website","name":"Cloudflare Docs","url":"https://developers.cloudflare.com/"},"keywords":["AI"]} ```