Website

The Website data source allows you to connect a domain you own so its pages can be crawled, stored, and indexed.

You can only crawl domains that you have onboarded onto the same Cloudflare account. Refer to Onboard a domain for more information on adding a domain to your Cloudflare account.

Bot protection may block crawling

If you use Cloudflare products that control or restrict bot traffic such as Bot Management, Web Application Firewall (WAF), or Turnstile, the same rules will apply to the AI Search (AutoRAG) crawler. Make sure to configure an exception or an allow-list for the AutoRAG crawler in your settings.

How website crawling works

When you connect a domain, the crawler looks for your website's sitemap to determine which pages to visit:

1. The crawler first checks the robots.txt for listed sitemaps. If it exists, it reads all sitemaps existing inside.
2. If no robots.txt is found, the crawler first checks for a sitemap at /sitemap.xml.
3. If no sitemap is available, the domain cannot be crawled.

Pages are visited, according to the <priority> attribute set on the sitemaps, if this field is defined.

How to set WAF rules to allowlist the crawler

If you have Security rules configured to block bot activity, you can add a rule to allowlist the crawler bot.

1. In the Cloudflare dashboard, go to the Security rules page.

   Go to Security rules

2. To create a new empty rule, select Create rule > Custom rules.

3. Enter a descriptive name for the rule in Rule name, such as Allow AI Search.

4. Under When incoming requests match, use the Field drop-down list to choose Bot Detection ID. For Operator, select equals. For Value, enter 122933950.

5. Under Then take action, in the Choose action dropdown, choose Skip.

6. Under Place at, select the order of the rule in the Select order dropdown to be First. Setting the order as First allows this rule to be applied before subsequent rules.

7. To save and deploy your rule, select Deploy.

Parsing options

You can choose how pages are parsed during crawling:

• Static sites: Downloads the raw HTML for each page.
• Rendered sites: Loads pages with a headless browser and downloads the fully rendered version, including dynamic JavaScript content. Note that the Browser Rendering limits and billing apply.

Access protected content

If your website has pages behind authentication or are only visible to logged-in users, you can configure custom HTTP headers to allow the AI Search crawler to access this protected content. You can add up to five custom HTTP headers to the requests AI Search sends when crawling your site.

Providing access to sites protected by Cloudflare Access

To allow AI Search to crawl a site protected by Cloudflare Access, you need to create service token credentials and configure them as custom headers.

Service tokens bypass user authentication, so ensure your Access policies are configured appropriately for the content you want to index. The service token will allow the AI Search crawler to access all content covered by the Service Auth policy.

1. In Cloudflare One ↗, create a service token. Once the Client ID and Client Secret are generated, save them for the next steps. For example they can look like:

   CF-Access-Client-Id: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.access
   CF-Access-Client-Secret: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

2. Create a policy with the following configuration:

   • Add an Include rule with Selector set to Service token.
   • In Value, select the Service Token you created in step 1.

3. Add your self-hosted application to Access and with the following configuration:

   • In Access policies, click Select existing policies.
   • Select the policy that you have just created and select Confirm.

4. In the Cloudflare dashboard, go to the AI Search page.

   Go to AI Search (AutoRAG)

5. Select Create.

6. Select Website as your data source.

7. Under Parse options, locate Extra headers and add the following two headers using your saved credentials:

   • Header 1:
     • Key: CF-Access-Client-Id
     • Value: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.access
   • Header 2:
     • Key: CF-Access-Client-Secret
     • Value: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

8. Complete the AI Search setup process to create your search instance.

Storage

During setup, AI Search creates a dedicated R2 bucket in your account to store the pages that have been crawled and downloaded as HTML files. This bucket is automatically managed and is used only for content discovered by the crawler. Any files or objects that you add directly to this bucket will not be indexed.

Note

We recommend not modifying the bucket as it may disrupt the indexing flow and cause content to not be updated properly.

Sync and updates

During scheduled or manual sync jobs, the crawler will check for changes to the <lastmod> attribute in your sitemap. If it has been changed to a date occurring after the last sync date, then the page will be crawled, the updated version is stored in the R2 bucket, and automatically reindexed so that your search results always reflect the latest content.

If the <lastmod> attribute is not defined, then AI Search will automatically crawl each link defined in the sitemap once a day.

Limits

The regular AI Search limits apply when using the Website data source.

The crawler will download and index pages only up to the maximum object limit supported for an AI Search instance, and it processes the first set of pages it visits until that limit is reached. In addition, any files that are downloaded but exceed the file size limit will not be indexed.