Parse types

The parse type controls how AI Search finds the pages to index on a website data source. AI Search supports two parse types.

Set the parse type in source_params.web_crawler.parse_type. If you do not set it, AI Search uses sitemap.

Both parse types require a source URL on a domain that you have onboarded onto the same Cloudflare account. Refer to Onboard a domain if your domain is not on Cloudflare yet.

Both parse types can read your sitemaps, so the choice is not simply "sitemap or links". discover defaults to a discovery source of all, which means it reads your sitemaps and follows links. What differs is how thoroughly each one uses the sitemap.

If your site publishes a sitemap that covers the pages you want indexed, prefer sitemap. It is the more reliable of the two, because:

• Updates are driven by your sitemap. sitemap re-crawls a page when its <lastmod> date changes, so edits are picked up on the next sync. discover ignores <lastmod> and <changefreq> entirely and re-fetches on a fixed cache age instead.
• Nothing is cut off. discover stops at the configured page limit and depth, so pages that sit deep in your link graph, or past the limit, can be skipped. sitemap indexes everything the sitemap lists.
• You control the order. sitemap indexes pages by their <priority> value, so your most important pages are indexed first if you hit an instance limit.
• You can narrow the crawl. Specific sitemap is not supported with discover.

Choose discover when a sitemap is missing, incomplete, or stale.

The sitemap parse type is the default. AI Search reads the XML sitemaps your site publishes to decide which pages to index, and re-crawls a page when its <lastmod> date changes.

When you connect a domain, the crawler looks for your website's sitemap to determine which pages to visit:

1. If you configure one or more custom sitemap URLs in the dashboard under Parser options > Specific sitemap, AI Search crawls only those sitemap URLs.
2. Otherwise, the crawler checks robots.txt for listed sitemaps.
3. If no robots.txt is found, the crawler checks for a sitemap at /sitemap.xml.
4. If no sitemap is available, the domain cannot be crawled with the sitemap parse type. Use discover instead.

If your sitemaps include <priority> attributes, AI Search reads all sitemaps and indexes pages based on each page's priority value, regardless of which sitemap the page is in.

If no <priority> is specified, pages are indexed in the order the sitemaps are provided, either from the configured custom sitemap URLs or from robots.txt from top to bottom.

AI Search supports .gz compressed sitemaps. Both robots.txt and sitemaps can use partial URLs.

During scheduled or manual sync jobs, the crawler will check for changes to the <lastmod> attribute in your sitemap. If it has been changed to a date occurring after the last sync date, then the page is crawled, the updated version is stored, and the page is automatically reindexed so that your search results always reflect the latest content.

If the <lastmod> attribute is not defined, AI Search uses the <changefreq> attribute to determine how often to re-crawl the URL. If neither <lastmod> nor <changefreq> is defined, AI Search automatically crawls each link once a day.

By default, AI Search crawls all sitemaps listed in your robots.txt in the order they appear (top to bottom). If you do not want the crawler to index everything, or if your sitemap is hosted at a non-standard path, you can configure custom sitemap URLs in the dashboard under Parser options > Specific sitemap.

When custom sitemap URLs are configured, AI Search uses those sitemap URLs instead of auto-discovering sitemaps from robots.txt or /sitemap.xml. You can add up to five sitemap URLs.

The AI Search crawler uses the user agent Cloudflare-AI-Search. Your robots.txt file should reference your sitemap and allow the crawler:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

You can list multiple sitemaps or use a sitemap index file:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/blog-sitemap.xml
Sitemap: https://example.com/sitemap.xml.gz

To block all other crawlers but allow only AI Search:

User-agent: *
Disallow: /

User-agent: Cloudflare-AI-Search
Allow: /

Sitemap: https://example.com/sitemap.xml

Structure your sitemap to give AI Search the information it needs to crawl efficiently:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/important-page</loc>
    <lastmod>2026-01-15</lastmod>
    <changefreq>weekly</changefreq>
    <priority>1.0</priority>
  </url>
  <url>
    <loc>https://example.com/other-page</loc>
    <lastmod>2026-01-10</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.5</priority>
  </url>
</urlset>

Use these attributes to control crawling behavior:

You can also use a Sitemap Index to bundle other domain-specific sitemaps:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://www.example.com/sitemap-blog.xml</loc>
    <lastmod>2024-08-15T10:00:00+00:00</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://www.example.com/sitemap-docs.xml</loc>
    <lastmod>2024-08-10T12:00:00+00:00</lastmod>
  </sitemap>
</sitemapindex>

When parsing a Sitemap Index, AI Search collects all child sitemaps and then crawls them recursively, collecting all relevant URLs present in your sitemaps.

• Include <lastmod> on all URLs to enable efficient change detection during syncs.
• Set <priority> to control indexing order. Pages with higher priority are indexed first.
• Use <changefreq> as a fallback when <lastmod> is not available.
• Use sitemap index files for large sites with multiple sitemaps.
• Compress large sitemaps using .gz format to reduce bandwidth.
• Keep sitemaps under 50MB and 50,000 URLs per file (standard sitemap limits).

The discover parse type delegates page discovery to the Browser Run /crawl endpoint. AI Search starts a crawl job at your source URL and stores every page it fetches in built-in storage.

By default, discover collects candidate URLs from both your sitemaps and the links on the pages it crawls. Use the discovery source option to restrict it to one or the other.

Because discover does not depend on a sitemap, it reaches pages that a sitemap omits.

1. AI Search starts a crawl job at the source URL.
2. The crawler collects candidate URLs from sitemaps, from links on crawled pages, or from both, depending on the discovery source.
3. The crawler follows links up to the configured depth and stops once it reaches the page limit.
4. Each fetched page is stored in built-in storage, converted to Markdown, chunked, and indexed.

The crawler identifies itself as Cloudflare-AI-Search on domains in your Cloudflare account. If you enable external links or subdomains and the crawl reaches a domain outside your account, it identifies itself as Cloudflare-AI-Search-External. Pages that robots.txt disallows are recorded with the blocked_by_robots_txt indexing error code instead of being indexed.

To change these options later, select your instance, open the Settings tab, and edit Crawl options under Parser options. Saving the change starts a new indexing job that reindexes every item.

Set parse_type to discover, then pass any non-default settings in discover_options.

curl -X POST "https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/ai-search/instances" \
  -H "Authorization: Bearer <API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "my-ai-search",
    "type": "web-crawler",
    "source": "example.com",
    "source_params": {
      "web_crawler": {
        "parse_type": "discover",
        "discover_options": {
          "source": "links",
          "limit": 5000,
          "depth": 3,
          "max_age": 86400,
          "include_subdomains": true
        }
      }
    }
  }'

The following options apply only when parse_type is discover. All of them are optional.

The source option selects where candidate URLs come from:

• all: Uses both sitemaps and links found on crawled pages.
• sitemaps: Uses only URLs listed in sitemaps.
• links: Uses only links found on crawled pages.

Use links when your sitemap is missing or unreliable. Use sitemaps when you want sitemap coverage without following in-page links.

The limit option caps the number of pages the crawler indexes, up to a maximum of 100000. Values above that maximum are rejected when you create or update the instance. Your instance object limit still applies, so the effective cap is whichever value is lower.

The depth option caps how far the crawler travels from the source URL. A depth of 1 crawls only the pages linked directly from the source URL.

The max_age option sets the maximum age, in seconds, of cached page content that the crawler accepts before it re-fetches the page from your origin. Set it to 0 to always fetch from the origin.

In the dashboard, Max cache age offers No cache, 1 hour, 1 day, 3 days, and 7 days.

By default, the crawler stays on the host of the source URL. Enable include_subdomains to follow links into subdomains, and include_external_links to follow links to other domains.

Each sync job starts a new crawl job. AI Search reindexes the pages the crawl returns, and removes pages it can no longer reach.

If a crawl does not finish, or if it stops at the page limit, AI Search keeps the pages it indexed previously rather than deleting the pages that crawl did not reach.

discover does not read the <lastmod> or <changefreq> sitemap attributes. Use max_age to control how often the crawler returns to your origin.

parse_options.specific_sitemaps is only valid when parse_type is sitemap. Sending it with discover returns a validation error.

The following website settings apply whichever parse type you choose:

• Path filtering: Include and exclude URL patterns. Excluded pages are never fetched.
• Content selectors: Restrict indexing to the elements a CSS selector matches.
• Authentication headers: Custom HTTP headers sent with each request.
• Custom metadata: Metadata extracted from <meta> tags.
• Rendering mode: Whether pages load in a headless browser.

A discover crawl indexes at most 100,000 pages and follows at most 100,000 link hops from the source URL, set with limit and depth.

The files per instance limit also applies, so the effective cap is whichever value is lower. For every limit that applies to website data sources, refer to Website.