Website

The Website data source allows you to connect a domain you own so its pages can be crawled, stored, and indexed.

Note

You can only crawl domains that you have onboarded onto the same Cloudflare account.

Refer to Onboard a domain for more information on adding a domain to your Cloudflare account.

How website crawling works

When you connect a domain, the crawler looks for your website’s sitemap to determine which pages to visit:

1. The crawler first checks the robots.txt for listed sitemaps. If it exists, it reads all sitemaps existing inside.
2. If no robots.txt is found, the crawler first checks for a sitemap at /sitemap.xml.
3. If no sitemap is available, the domain cannot be crawled.

Pages are visited, according to the <priority> attribute set on the sitemaps, if this field is defined.

Parsing options

You can choose how pages are parsed during crawling:

• Static sites: Downloads the raw HTML for each page.
• Rendered sites: Loads pages with a headless browser and downloads the fully rendered version, including dynamic JavaScript content. Note that the Browser Rendering limits and billing apply.

Storage

During setup, AutoRAG creates a dedicated R2 bucket in your account to store the pages that have been crawled and downloaded as HTML files. This bucket is automatically managed and is used only for content discovered by the crawler. Any files or objects that you add directly to this bucket will not be indexed.

Note

We recommend not to modify the bucket as it may distrupt the indexing flow and cause content to not be updated properly.

Sync and updates

During scheduled or manual sync jobs, the crawler will check for changes to the <lastmod> attribute in your sitemap. If it has been changed to a date occuring after the last sync date, then the page will be crawled, the updated version is stored in the R2 bucket, and automatically reindexed so that your search results always reflect the latest content.

If the <lastmod> attribute is not defined, then AutoRAG will automatically crawl each link defined in the sitemap once a day.

Limits

The regular AutoRAG limits apply when using the Website data source.

The crawler will download and index pages only up to the maximum object limit supported for an AutoRAG instance, and it processes the first set of pages it visits until that limit is reached. In addition, any files that are downloaded but exceed the file size limit will not be indexed.