Skip to content

Authentication headers

Last updated View as MarkdownAgent setup

You can only crawl domains that you have onboarded onto the same Cloudflare account. Refer to Onboard a domain for more information on adding a domain to your Cloudflare account.

If your website has pages behind authentication or pages that are only visible to logged-in users, you can configure custom HTTP headers to allow the AI Search crawler to access this protected content. You can add up to five custom HTTP headers to the requests AI Search sends when crawling your site.

This setting is labeled Extra headers in the dashboard, under Parser options. In the REST API and in Wrangler, it is the source_params.web_crawler.parse_options.include_headers field.

Configure in the dashboard

  1. In the Cloudflare dashboard, go to the AI Search page.

    Go to AI Search ↗

  2. Select Create, then select Website as your data source. To add headers to an existing instance, select the instance and open the Settings tab.

  3. Under Parser options, locate Extra headers.

  4. Add a header, entering the header name in Key and the header value in Value. For example, Key Authorization and Value Bearer <TOKEN>.

  5. Repeat for each header you need, up to five.

  6. Save your changes.

The crawler sends every header you configure with each request it makes to your site. Adding or changing headers on an existing instance starts a new indexing job that reindexes every item.

Indexing your site protected by Cloudflare Access

To allow AI Search to crawl a site protected by Cloudflare Access, you need to create service token credentials and configure them as custom headers.

Service tokens bypass user authentication, so ensure your Access policies are configured appropriately for the content you want to index. The service token will allow the AI Search crawler to access all content covered by the Service Auth policy.

  1. In the Cloudflare dashboard ↗, create a service token. Once the Client ID and Client Secret are generated, save them for the next steps. For example they can look like:

    CF-Access-Client-Id: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.access
CF-Access-Client-Secret: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

  2. Create a policy with the following configuration:

    • Add an Include rule with Selector set to Service token.
    • In Value, select the Service Token you created in step 1.

  3. Add your self-hosted application to Access with the following configuration:

    • In Access policies, click Select existing policies.
    • Select the policy that you have just created and select Confirm.

  4. In the Cloudflare dashboard, go to the AI Search page.

    Go to AI Search ↗

  5. Select Create.

  6. Select Website as your data source.

  7. Under Parser options, locate Extra headers and add the following two headers using your saved credentials:

    • Header 1:
      • Key: CF-Access-Client-Id
      • Value: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.access
    • Header 2:
      • Key: CF-Access-Client-Secret
      • Value: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

  8. Complete the AI Search setup process to create your search instance.

Was this helpful?