--- title: Overview description: Cloudflare AI Search is a managed search service. Index your content and query it with natural language from a Workers binding, REST API, or MCP server. image: https://developers.cloudflare.com/dev-products-preview.png --- > Documentation Index > Fetch the complete documentation index at: https://developers.cloudflare.com/ai-search/llms.txt > Use this file to discover all available pages before exploring further. [Skip to content](#%5Ftop) # Overview The search primitive for your applications and agents. Available on all plans AI Search lets you add search to any application or agent without having to build an entire retrieval infrastructure. Create an instance, give it your data, and search it with natural language. You can use AI Search for: * Documentation and knowledge base search * AI agent tool use and memory * Per-tenant or per-agent file search [ Get started ](https://developers.cloudflare.com/ai-search/get-started/)[ Watch AI Search demo ](https://www.youtube.com/watch?v=JUFdbkiDN2U) Latest update New AI Search instances created after April 16, 2026 include [managed storage](https://developers.cloudflare.com/ai-search/configuration/data-source/built-in-storage/), vector index, and web crawling. [View limits and pricing](https://developers.cloudflare.com/ai-search/platform/limits-pricing/). --- ## Features ### Automated indexing Automatically and continuously index your data source, keeping your content fresh without manual reprocessing. [ View indexing ](https://developers.cloudflare.com/ai-search/configuration/indexing/syncing/) ### Metadata filtering Define custom metadata fields and filter search results by category, version, language, or any attribute you define. [ Add filters ](https://developers.cloudflare.com/ai-search/configuration/indexing/metadata/) ### Hybrid search Combine semantic and keyword matching in the same query for more accurate results. [ Configure hybrid search ](https://developers.cloudflare.com/ai-search/configuration/indexing/hybrid-search/) ### MCP and UI snippets Every instance includes a built-in MCP endpoint for AI agents and embeddable search components for your website. [ Connect agents ](https://developers.cloudflare.com/ai-search/api/search/mcp/) --- ## Related products **[Workers AI](https://developers.cloudflare.com/workers-ai/)** Run machine learning models, powered by serverless GPUs, on Cloudflare's global network. **[AI Gateway](https://developers.cloudflare.com/ai-gateway/)** Observe and control your AI applications with caching, rate limiting, request retries, model fallback, and more. **[Vectorize](https://developers.cloudflare.com/vectorize/)** Build full-stack AI applications with Vectorize, Cloudflare's vector database. **[Workers](https://developers.cloudflare.com/workers/)** Build serverless applications and deploy instantly across the globe for exceptional performance, reliability, and scale. **[R2](https://developers.cloudflare.com/r2/)** Store large amounts of unstructured data without the costly egress bandwidth fees associated with typical cloud storage services. --- ## More resources [Get started](https://developers.cloudflare.com/ai-search/get-started/) Create your first AI Search instance and run your first query. [Developer Discord](https://discord.cloudflare.com) Connect with the Workers community on Discord to ask questions, share what you are building, and discuss the platform with other developers. [@CloudflareDev](https://x.com/cloudflaredev) Follow @CloudflareDev on Twitter to learn about product announcements, and what is new in Cloudflare Workers. ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/ai-search/","name":"AI Search"}}]} ``` --- --- title: Get started description: Create fully-managed, retrieval-augmented generation pipelines with Cloudflare AI Search. image: https://developers.cloudflare.com/dev-products-preview.png --- > Documentation Index > Fetch the complete documentation index at: https://developers.cloudflare.com/ai-search/llms.txt > Use this file to discover all available pages before exploring further. [Skip to content](#%5Ftop) # Get started AI Search is a managed search service. Connect a website, an R2 bucket, or upload your own documents, and AI Search indexes your content for natural language queries. ## Prerequisites If you plan to use an R2 bucket as your data source, you must have an active [R2 subscription](https://developers.cloudflare.com/r2/get-started/) before creating your AI Search instance. If you plan to use a website or upload files directly, no additional setup is required. ## Choose your setup method [ Dashboard ](https://developers.cloudflare.com/ai-search/get-started/dashboard/) Create and configure AI Search using the Cloudflare dashboard. [ API ](https://developers.cloudflare.com/ai-search/get-started/api/) Create AI Search instances programmatically using the REST API. [ Wrangler commands ](https://developers.cloudflare.com/ai-search/get-started/wrangler/) Create and manage AI Search instances from the command line. ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/ai-search/","name":"AI Search"}},{"@type":"ListItem","position":3,"item":{"@id":"/ai-search/get-started/","name":"Get started"}}]} ``` --- --- title: REST API description: Create AI Search instances programmatically using the REST API. image: https://developers.cloudflare.com/dev-products-preview.png --- > Documentation Index > Fetch the complete documentation index at: https://developers.cloudflare.com/ai-search/llms.txt > Use this file to discover all available pages before exploring further. [Skip to content](#%5Ftop) # REST API This guide walks you through creating an AI Search instance using the REST API. ## 1\. Create an API token You need an API token with **AI Search:Edit** and **AI Search:Run** permissions. 1. In the Cloudflare dashboard, go to **My Profile** \> **API Tokens**. [ Go to **API Tokens** ](https://dash.cloudflare.com/profile/api-tokens) 2. Select **Create Token**. 3. Select **Create Custom Token**. 4. Enter a **Token name**, for example `AI Search Manager`. 5. Under **Permissions**, add two permissions: * **Account** \> **AI Search:Edit** * **Account** \> **AI Search:Run** 6. Select **Continue to summary**, then select **Create Token**. 7. Copy and save the token value. This is your `API_TOKEN`. ## 2\. Create an AI Search instance Use the [Create instance API](https://developers.cloudflare.com/api/resources/ai%5Fsearch/subresources/instances/methods/create/) to create an instance. Replace `` with your [account ID](https://developers.cloudflare.com/fundamentals/account/find-account-and-zone-ids/). Terminal window ``` curl -X POST "https://api.cloudflare.com/client/v4/accounts//ai-search/instances" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ --data '{ "id": "my-instance" }' ``` ### Connect a data source (optional) You can create an instance that is connected to a website or R2 bucket as a data source. AI Search indexes the content automatically. **Website:** Automatically crawl and index a [website](https://developers.cloudflare.com/ai-search/configuration/data-source/website/) that you own. Terminal window ``` curl -X POST "https://api.cloudflare.com/client/v4/accounts//ai-search/instances" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ --data '{ "id": "my-instance", "type": "web-crawler", "source": "example.com" }' ``` **R2 bucket:** Index documents stored in an [R2 bucket](https://developers.cloudflare.com/ai-search/configuration/data-source/r2/). Connecting an R2 bucket requires a [service API token](https://developers.cloudflare.com/ai-search/configuration/indexing/service-api-token/). If you have never created an R2-backed instance before, you need to pass the `token_id` field in the create request. Refer to the [service API token configuration](https://developers.cloudflare.com/ai-search/configuration/indexing/service-api-token/) for setup instructions. Terminal window ``` curl -X POST "https://api.cloudflare.com/client/v4/accounts//ai-search/instances" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ --data '{ "id": "my-instance", "type": "r2", "source": "", "token_id": "" }' ``` ## 3\. Add content If you did not create an instance that is connected to a data source, upload files using the [Items API](https://developers.cloudflare.com/api/resources/ai%5Fsearch/subresources/namespaces/subresources/instances/subresources/items/methods/upload/). You can skip this step if you connected a website or R2 bucket. Terminal window ``` curl -X POST "https://api.cloudflare.com/client/v4/accounts//ai-search/instances/my-instance/items" \ -H "Authorization: Bearer " \ -F "file=@/path/to/your/file.pdf" ``` AI Search indexes uploaded files automatically. ## 4\. Check indexing status Check if your content has finished indexing. Terminal window ``` curl "https://api.cloudflare.com/client/v4/accounts//ai-search/instances/my-instance/stats" \ -H "Authorization: Bearer " ``` ## Try it out Once indexing is complete, run your first query. Terminal window ``` curl "https://api.cloudflare.com/client/v4/accounts//ai-search/instances/my-instance/search" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "messages": [ { "content": "How do I get started?", "role": "user" } ] }' ``` You can also test queries in the dashboard by going to your instance and selecting the **Playground** tab. ## Add to your application [ Workers binding ](https://developers.cloudflare.com/ai-search/api/search/workers-binding/) Query AI Search directly from your Workers code. [ REST API ](https://developers.cloudflare.com/ai-search/api/search/rest-api/) Query AI Search using HTTP requests. ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/ai-search/","name":"AI Search"}},{"@type":"ListItem","position":3,"item":{"@id":"/ai-search/get-started/","name":"Get started"}},{"@type":"ListItem","position":4,"item":{"@id":"/ai-search/get-started/api/","name":"REST API"}}]} ``` --- --- title: Dashboard description: Create and configure AI Search using the Cloudflare dashboard. image: https://developers.cloudflare.com/dev-products-preview.png --- > Documentation Index > Fetch the complete documentation index at: https://developers.cloudflare.com/ai-search/llms.txt > Use this file to discover all available pages before exploring further. [Skip to content](#%5Ftop) # Dashboard This guide walks you through creating an AI Search instance using the Cloudflare dashboard. ## Create an AI Search instance 1. Go to **AI Search** in the Cloudflare dashboard. [ Go to **AI Search** ](https://dash.cloudflare.com/?to=/:account/ai/ai-search) 2. Select **Create Instance**. 3. Name your AI Search instance. 4. Optionally connect a [data source](https://developers.cloudflare.com/ai-search/configuration/data-source/) such as a website you own or an R2 bucket. 5. Review your configurations. 6. Select **Create**. ## Upload content Once an instance has been created, you can upload files directly from the dashboard. 1. Go to **AI Search** in the Cloudflare dashboard. [ Go to **AI Search** ](https://dash.cloudflare.com/?to=/:account/ai/ai-search) 2. Select your instance. 3. Select the **Items** tab. 4. Upload your files. AI Search indexes them automatically. ## Try it out Once your content is indexed, you can run your first query. 1. Go to your AI Search instance. 2. Select the **Playground** tab. 3. Select **Chat** or **Search**. 4. Enter a query to test the response. ## Add to your application There are multiple ways you can connect AI Search to your application: [ Workers Binding ](https://developers.cloudflare.com/ai-search/api/search/workers-binding/) Query AI Search directly from your Workers code. [ REST API ](https://developers.cloudflare.com/ai-search/api/search/rest-api/) Query AI Search using HTTP requests. ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/ai-search/","name":"AI Search"}},{"@type":"ListItem","position":3,"item":{"@id":"/ai-search/get-started/","name":"Get started"}},{"@type":"ListItem","position":4,"item":{"@id":"/ai-search/get-started/dashboard/","name":"Dashboard"}}]} ``` --- --- title: Wrangler commands description: Create and manage AI Search instances from the command line. image: https://developers.cloudflare.com/dev-products-preview.png --- > Documentation Index > Fetch the complete documentation index at: https://developers.cloudflare.com/ai-search/llms.txt > Use this file to discover all available pages before exploring further. [Skip to content](#%5Ftop) # Wrangler commands This guide walks you through creating an AI Search instance using the [Wrangler CLI](https://developers.cloudflare.com/workers/wrangler/). ## 1\. Install Wrangler Install [Wrangler](https://developers.cloudflare.com/workers/wrangler/), the command-line tool for Cloudflare Workers and developer platform products. npm yarn pnpm bun ``` npm install wrangler ``` ``` yarn install wrangler ``` ``` pnpm install wrangler ``` ``` bun install wrangler ``` ## 2\. Create an AI Search instance Create a new instance. Terminal window ``` wrangler ai-search create my-instance ``` You can upload files to the instance using the [dashboard](https://developers.cloudflare.com/ai-search/get-started/dashboard/#upload-content) or the [REST API](https://developers.cloudflare.com/api/resources/ai%5Fsearch/subresources/namespaces/subresources/instances/subresources/items/methods/upload/). ### Connect a data source (optional) You can optionally connect a website or R2 bucket when creating the instance. **Website:** Automatically crawl and index a [website](https://developers.cloudflare.com/ai-search/configuration/data-source/website/) that you own. Terminal window ``` wrangler ai-search create my-instance --type web-crawler --source developers.cloudflare.com ``` **R2 bucket:** Index documents stored in an [R2 bucket](https://developers.cloudflare.com/ai-search/configuration/data-source/r2/). Terminal window ``` wrangler ai-search create my-instance --type r2 --source my-bucket ``` ## 3\. Check indexing status Check if your content has finished indexing by running the `stats` command. Terminal window ``` wrangler ai-search stats my-instance ``` ## 4\. Test your instance Once indexing is complete, run a search query against your instance. Terminal window ``` wrangler ai-search search my-instance --query "What is Cloudflare?" ``` For the full list of available commands, refer to [Wrangler commands](https://developers.cloudflare.com/ai-search/wrangler-commands/). ## Add to your application [ Workers binding ](https://developers.cloudflare.com/ai-search/api/search/workers-binding/) Query AI Search directly from your Workers code. [ REST API ](https://developers.cloudflare.com/ai-search/api/search/rest-api/) Query AI Search using HTTP requests. ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/ai-search/","name":"AI Search"}},{"@type":"ListItem","position":3,"item":{"@id":"/ai-search/get-started/","name":"Get started"}},{"@type":"ListItem","position":4,"item":{"@id":"/ai-search/get-started/wrangler/","name":"Wrangler commands"}}]} ``` --- --- title: Wrangler commands description: Manage AI Search instances from the command line using Wrangler. image: https://developers.cloudflare.com/dev-products-preview.png --- > Documentation Index > Fetch the complete documentation index at: https://developers.cloudflare.com/ai-search/llms.txt > Use this file to discover all available pages before exploring further. [Skip to content](#%5Ftop) # Wrangler commands ## `ai-search list` List all AI Search instances * [ npm ](#tab-panel-6489) * [ pnpm ](#tab-panel-6490) * [ yarn ](#tab-panel-6491) Terminal window ``` npx wrangler ai-search list ``` Terminal window ``` pnpm wrangler ai-search list ``` Terminal window ``` yarn wrangler ai-search list ``` * `--namespace` ` string ` alias: --n default: default The namespace to list instances from. * `--json` ` boolean ` default: false Return output as clean JSON * `--page` ` number ` default: 1 Page number of the results, can configure page size using "per-page" * `--per-page` ` number ` Number of instances to show per page Global flags * `--v` ` boolean ` alias: --version Show version number * `--cwd` ` string ` Run as if Wrangler was started in the specified directory instead of the current working directory * `--config` ` string ` alias: --c Path to Wrangler configuration file * `--env` ` string ` alias: --e Environment to use for operations, and for selecting .env and .dev.vars files * `--env-file` ` string ` Path to an .env file to load - can be specified multiple times - values from earlier files are overridden by values in later files * `--experimental-provision` ` boolean ` aliases: --x-provision default: true Experimental: Enable automatic resource provisioning * `--experimental-auto-create` ` boolean ` alias: --x-auto-create default: true Automatically provision draft bindings with new resources * `--install-skills` ` boolean ` default: false Install Cloudflare agents skills, if not already present, without asking the user for confirmation ## `ai-search create` Create a new AI Search instance * [ npm ](#tab-panel-6492) * [ pnpm ](#tab-panel-6493) * [ yarn ](#tab-panel-6494) Terminal window ``` npx wrangler ai-search create [NAME] ``` Terminal window ``` pnpm wrangler ai-search create [NAME] ``` Terminal window ``` yarn wrangler ai-search create [NAME] ``` * `[NAME]` ` string ` required The name of the AI Search instance to create (must be unique within its namespace). * `--namespace` ` string ` alias: --n The namespace to create the instance in. * `--source` ` string ` Data source identifier (R2 bucket name or web URL). * `--type` ` string ` The source type for the instance. * `--embedding-model` ` string ` Embedding model to use. * `--generation-model` ` string ` LLM model for chat completions. * `--chunk-size` ` number ` Chunk size for document splitting (min: 64). * `--chunk-overlap` ` number ` Overlap between document chunks. * `--max-num-results` ` number ` Maximum search results per query. * `--reranking` ` boolean ` Enable reranking of search results. * `--reranking-model` ` string ` Model to use for reranking. * `--hybrid-search` ` boolean ` Enable hybrid (keyword + vector) search. * `--cache` ` boolean ` Enable response caching. * `--score-threshold` ` number ` Minimum relevance score threshold (0-1). * `--prefix` ` string ` R2 key prefix to scope indexing. * `--include-items` ` array ` Glob patterns for items to include. * `--exclude-items` ` array ` Glob patterns for items to exclude. * `--custom-metadata` ` array ` Custom metadata fields, formatted as 'field\_name:data\_type'. data\_type must be one of: text, number, boolean, datetime. Repeat the flag for multiple fields (e.g. --custom-metadata title:text --custom-metadata views:number). * `--custom-metadata-schema` ` string ` Path to a JSON file describing custom metadata fields. The file must contain an array of { "field\_name", "data\_type" } objects. Mutually exclusive with --custom-metadata. * `--json` ` boolean ` default: false Return output as clean JSON Global flags * `--v` ` boolean ` alias: --version Show version number * `--cwd` ` string ` Run as if Wrangler was started in the specified directory instead of the current working directory * `--config` ` string ` alias: --c Path to Wrangler configuration file * `--env` ` string ` alias: --e Environment to use for operations, and for selecting .env and .dev.vars files * `--env-file` ` string ` Path to an .env file to load - can be specified multiple times - values from earlier files are overridden by values in later files * `--experimental-provision` ` boolean ` aliases: --x-provision default: true Experimental: Enable automatic resource provisioning * `--experimental-auto-create` ` boolean ` alias: --x-auto-create default: true Automatically provision draft bindings with new resources * `--install-skills` ` boolean ` default: false Install Cloudflare agents skills, if not already present, without asking the user for confirmation ## `ai-search get` Get details of an AI Search instance * [ npm ](#tab-panel-6495) * [ pnpm ](#tab-panel-6496) * [ yarn ](#tab-panel-6497) Terminal window ``` npx wrangler ai-search get [NAME] ``` Terminal window ``` pnpm wrangler ai-search get [NAME] ``` Terminal window ``` yarn wrangler ai-search get [NAME] ``` * `[NAME]` ` string ` required The name of the AI Search instance. * `--namespace` ` string ` alias: --n default: default The namespace the instance belongs to. * `--json` ` boolean ` default: false Return output as clean JSON Global flags * `--v` ` boolean ` alias: --version Show version number * `--cwd` ` string ` Run as if Wrangler was started in the specified directory instead of the current working directory * `--config` ` string ` alias: --c Path to Wrangler configuration file * `--env` ` string ` alias: --e Environment to use for operations, and for selecting .env and .dev.vars files * `--env-file` ` string ` Path to an .env file to load - can be specified multiple times - values from earlier files are overridden by values in later files * `--experimental-provision` ` boolean ` aliases: --x-provision default: true Experimental: Enable automatic resource provisioning * `--experimental-auto-create` ` boolean ` alias: --x-auto-create default: true Automatically provision draft bindings with new resources * `--install-skills` ` boolean ` default: false Install Cloudflare agents skills, if not already present, without asking the user for confirmation ## `ai-search update` Update an AI Search instance configuration * [ npm ](#tab-panel-6498) * [ pnpm ](#tab-panel-6499) * [ yarn ](#tab-panel-6500) Terminal window ``` npx wrangler ai-search update [NAME] ``` Terminal window ``` pnpm wrangler ai-search update [NAME] ``` Terminal window ``` yarn wrangler ai-search update [NAME] ``` * `[NAME]` ` string ` required The name of the AI Search instance to update. * `--namespace` ` string ` alias: --n default: default The namespace the instance belongs to. * `--embedding-model` ` string ` Update the embedding model. * `--generation-model` ` string ` Update the LLM model for chat completions. * `--chunk-size` ` number ` Update the chunk size. * `--chunk-overlap` ` number ` Update the chunk overlap. * `--max-num-results` ` number ` Update max search results per query. * `--reranking` ` boolean ` Enable or disable reranking. * `--reranking-model` ` string ` Update the reranking model. * `--hybrid-search` ` boolean ` Enable or disable hybrid search. * `--cache` ` boolean ` Enable or disable caching. * `--score-threshold` ` number ` Update the minimum relevance score threshold (0-1). * `--paused` ` boolean ` Pause or resume the instance. * `--json` ` boolean ` default: false Return output as clean JSON Global flags * `--v` ` boolean ` alias: --version Show version number * `--cwd` ` string ` Run as if Wrangler was started in the specified directory instead of the current working directory * `--config` ` string ` alias: --c Path to Wrangler configuration file * `--env` ` string ` alias: --e Environment to use for operations, and for selecting .env and .dev.vars files * `--env-file` ` string ` Path to an .env file to load - can be specified multiple times - values from earlier files are overridden by values in later files * `--experimental-provision` ` boolean ` aliases: --x-provision default: true Experimental: Enable automatic resource provisioning * `--experimental-auto-create` ` boolean ` alias: --x-auto-create default: true Automatically provision draft bindings with new resources * `--install-skills` ` boolean ` default: false Install Cloudflare agents skills, if not already present, without asking the user for confirmation ## `ai-search delete` Delete an AI Search instance * [ npm ](#tab-panel-6501) * [ pnpm ](#tab-panel-6502) * [ yarn ](#tab-panel-6503) Terminal window ``` npx wrangler ai-search delete [NAME] ``` Terminal window ``` pnpm wrangler ai-search delete [NAME] ``` Terminal window ``` yarn wrangler ai-search delete [NAME] ``` * `[NAME]` ` string ` required The name of the AI Search instance to delete. * `--namespace` ` string ` alias: --n default: default The namespace the instance belongs to. * `--force` ` boolean ` alias: --y default: false Skip confirmation Global flags * `--v` ` boolean ` alias: --version Show version number * `--cwd` ` string ` Run as if Wrangler was started in the specified directory instead of the current working directory * `--config` ` string ` alias: --c Path to Wrangler configuration file * `--env` ` string ` alias: --e Environment to use for operations, and for selecting .env and .dev.vars files * `--env-file` ` string ` Path to an .env file to load - can be specified multiple times - values from earlier files are overridden by values in later files * `--experimental-provision` ` boolean ` aliases: --x-provision default: true Experimental: Enable automatic resource provisioning * `--experimental-auto-create` ` boolean ` alias: --x-auto-create default: true Automatically provision draft bindings with new resources * `--install-skills` ` boolean ` default: false Install Cloudflare agents skills, if not already present, without asking the user for confirmation ## `ai-search stats` Get usage statistics for an AI Search instance * [ npm ](#tab-panel-6504) * [ pnpm ](#tab-panel-6505) * [ yarn ](#tab-panel-6506) Terminal window ``` npx wrangler ai-search stats [NAME] ``` Terminal window ``` pnpm wrangler ai-search stats [NAME] ``` Terminal window ``` yarn wrangler ai-search stats [NAME] ``` * `[NAME]` ` string ` required The name of the AI Search instance. * `--namespace` ` string ` alias: --n default: default The namespace the instance belongs to. * `--json` ` boolean ` default: false Return output as clean JSON Global flags * `--v` ` boolean ` alias: --version Show version number * `--cwd` ` string ` Run as if Wrangler was started in the specified directory instead of the current working directory * `--config` ` string ` alias: --c Path to Wrangler configuration file * `--env` ` string ` alias: --e Environment to use for operations, and for selecting .env and .dev.vars files * `--env-file` ` string ` Path to an .env file to load - can be specified multiple times - values from earlier files are overridden by values in later files * `--experimental-provision` ` boolean ` aliases: --x-provision default: true Experimental: Enable automatic resource provisioning * `--experimental-auto-create` ` boolean ` alias: --x-auto-create default: true Automatically provision draft bindings with new resources * `--install-skills` ` boolean ` default: false Install Cloudflare agents skills, if not already present, without asking the user for confirmation ## `ai-search search` Execute a semantic search query against an AI Search instance * [ npm ](#tab-panel-6507) * [ pnpm ](#tab-panel-6508) * [ yarn ](#tab-panel-6509) Terminal window ``` npx wrangler ai-search search [NAME] ``` Terminal window ``` pnpm wrangler ai-search search [NAME] ``` Terminal window ``` yarn wrangler ai-search search [NAME] ``` * `[NAME]` ` string ` required The name of the AI Search instance. * `--namespace` ` string ` alias: --n default: default The namespace the instance belongs to. * `--query` ` string ` required The search query text. * `--max-num-results` ` number ` Override maximum number of results. * `--score-threshold` ` number ` Override minimum relevance score (0-1). * `--reranking` ` boolean ` Override reranking setting. * `--filter` ` array ` Metadata filter as key=value (repeatable, e.g. --filter type=docs --filter lang=en). * `--json` ` boolean ` default: false Return output as clean JSON Global flags * `--v` ` boolean ` alias: --version Show version number * `--cwd` ` string ` Run as if Wrangler was started in the specified directory instead of the current working directory * `--config` ` string ` alias: --c Path to Wrangler configuration file * `--env` ` string ` alias: --e Environment to use for operations, and for selecting .env and .dev.vars files * `--env-file` ` string ` Path to an .env file to load - can be specified multiple times - values from earlier files are overridden by values in later files * `--experimental-provision` ` boolean ` aliases: --x-provision default: true Experimental: Enable automatic resource provisioning * `--experimental-auto-create` ` boolean ` alias: --x-auto-create default: true Automatically provision draft bindings with new resources * `--install-skills` ` boolean ` default: false Install Cloudflare agents skills, if not already present, without asking the user for confirmation ## `ai-search namespace list` List all AI Search namespaces * [ npm ](#tab-panel-6510) * [ pnpm ](#tab-panel-6511) * [ yarn ](#tab-panel-6512) Terminal window ``` npx wrangler ai-search namespace list ``` Terminal window ``` pnpm wrangler ai-search namespace list ``` Terminal window ``` yarn wrangler ai-search namespace list ``` * `--json` ` boolean ` default: false Return output as clean JSON * `--page` ` number ` default: 1 Page number of the results, can configure page size using "per-page" * `--per-page` ` number ` Number of namespaces to show per page * `--search` ` string ` Filter namespaces whose name or description contains this string (case-insensitive). Global flags * `--v` ` boolean ` alias: --version Show version number * `--cwd` ` string ` Run as if Wrangler was started in the specified directory instead of the current working directory * `--config` ` string ` alias: --c Path to Wrangler configuration file * `--env` ` string ` alias: --e Environment to use for operations, and for selecting .env and .dev.vars files * `--env-file` ` string ` Path to an .env file to load - can be specified multiple times - values from earlier files are overridden by values in later files * `--experimental-provision` ` boolean ` aliases: --x-provision default: true Experimental: Enable automatic resource provisioning * `--experimental-auto-create` ` boolean ` alias: --x-auto-create default: true Automatically provision draft bindings with new resources * `--install-skills` ` boolean ` default: false Install Cloudflare agents skills, if not already present, without asking the user for confirmation ## `ai-search namespace create` Create a new AI Search namespace * [ npm ](#tab-panel-6513) * [ pnpm ](#tab-panel-6514) * [ yarn ](#tab-panel-6515) Terminal window ``` npx wrangler ai-search namespace create [NAME] ``` Terminal window ``` pnpm wrangler ai-search namespace create [NAME] ``` Terminal window ``` yarn wrangler ai-search namespace create [NAME] ``` * `[NAME]` ` string ` required The name of the AI Search namespace to create. * `--description` ` string ` Optional description for the namespace (max 256 chars). * `--json` ` boolean ` default: false Return output as clean JSON Global flags * `--v` ` boolean ` alias: --version Show version number * `--cwd` ` string ` Run as if Wrangler was started in the specified directory instead of the current working directory * `--config` ` string ` alias: --c Path to Wrangler configuration file * `--env` ` string ` alias: --e Environment to use for operations, and for selecting .env and .dev.vars files * `--env-file` ` string ` Path to an .env file to load - can be specified multiple times - values from earlier files are overridden by values in later files * `--experimental-provision` ` boolean ` aliases: --x-provision default: true Experimental: Enable automatic resource provisioning * `--experimental-auto-create` ` boolean ` alias: --x-auto-create default: true Automatically provision draft bindings with new resources * `--install-skills` ` boolean ` default: false Install Cloudflare agents skills, if not already present, without asking the user for confirmation ## `ai-search namespace get` Get details of an AI Search namespace * [ npm ](#tab-panel-6516) * [ pnpm ](#tab-panel-6517) * [ yarn ](#tab-panel-6518) Terminal window ``` npx wrangler ai-search namespace get [NAME] ``` Terminal window ``` pnpm wrangler ai-search namespace get [NAME] ``` Terminal window ``` yarn wrangler ai-search namespace get [NAME] ``` * `[NAME]` ` string ` required The name of the AI Search namespace. * `--json` ` boolean ` default: false Return output as clean JSON Global flags * `--v` ` boolean ` alias: --version Show version number * `--cwd` ` string ` Run as if Wrangler was started in the specified directory instead of the current working directory * `--config` ` string ` alias: --c Path to Wrangler configuration file * `--env` ` string ` alias: --e Environment to use for operations, and for selecting .env and .dev.vars files * `--env-file` ` string ` Path to an .env file to load - can be specified multiple times - values from earlier files are overridden by values in later files * `--experimental-provision` ` boolean ` aliases: --x-provision default: true Experimental: Enable automatic resource provisioning * `--experimental-auto-create` ` boolean ` alias: --x-auto-create default: true Automatically provision draft bindings with new resources * `--install-skills` ` boolean ` default: false Install Cloudflare agents skills, if not already present, without asking the user for confirmation ## `ai-search namespace update` Update an AI Search namespace * [ npm ](#tab-panel-6519) * [ pnpm ](#tab-panel-6520) * [ yarn ](#tab-panel-6521) Terminal window ``` npx wrangler ai-search namespace update [NAME] ``` Terminal window ``` pnpm wrangler ai-search namespace update [NAME] ``` Terminal window ``` yarn wrangler ai-search namespace update [NAME] ``` * `[NAME]` ` string ` required The name of the AI Search namespace to update. * `--description` ` string ` Updated description for the namespace (max 256 chars). * `--json` ` boolean ` default: false Return output as clean JSON Global flags * `--v` ` boolean ` alias: --version Show version number * `--cwd` ` string ` Run as if Wrangler was started in the specified directory instead of the current working directory * `--config` ` string ` alias: --c Path to Wrangler configuration file * `--env` ` string ` alias: --e Environment to use for operations, and for selecting .env and .dev.vars files * `--env-file` ` string ` Path to an .env file to load - can be specified multiple times - values from earlier files are overridden by values in later files * `--experimental-provision` ` boolean ` aliases: --x-provision default: true Experimental: Enable automatic resource provisioning * `--experimental-auto-create` ` boolean ` alias: --x-auto-create default: true Automatically provision draft bindings with new resources * `--install-skills` ` boolean ` default: false Install Cloudflare agents skills, if not already present, without asking the user for confirmation ## `ai-search namespace delete` Delete an AI Search namespace * [ npm ](#tab-panel-6522) * [ pnpm ](#tab-panel-6523) * [ yarn ](#tab-panel-6524) Terminal window ``` npx wrangler ai-search namespace delete [NAME] ``` Terminal window ``` pnpm wrangler ai-search namespace delete [NAME] ``` Terminal window ``` yarn wrangler ai-search namespace delete [NAME] ``` * `[NAME]` ` string ` required The name of the AI Search namespace to delete. * `--force` ` boolean ` alias: --y default: false Skip confirmation Global flags * `--v` ` boolean ` alias: --version Show version number * `--cwd` ` string ` Run as if Wrangler was started in the specified directory instead of the current working directory * `--config` ` string ` alias: --c Path to Wrangler configuration file * `--env` ` string ` alias: --e Environment to use for operations, and for selecting .env and .dev.vars files * `--env-file` ` string ` Path to an .env file to load - can be specified multiple times - values from earlier files are overridden by values in later files * `--experimental-provision` ` boolean ` aliases: --x-provision default: true Experimental: Enable automatic resource provisioning * `--experimental-auto-create` ` boolean ` alias: --x-auto-create default: true Automatically provision draft bindings with new resources * `--install-skills` ` boolean ` default: false Install Cloudflare agents skills, if not already present, without asking the user for confirmation ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/ai-search/","name":"AI Search"}},{"@type":"ListItem","position":3,"item":{"@id":"/ai-search/wrangler-commands/","name":"Wrangler commands"}}]} ``` --- --- title: Configuration description: Customize how your AI Search instance indexes data, retrieves results, and generates responses. image: https://developers.cloudflare.com/dev-products-preview.png --- > Documentation Index > Fetch the complete documentation index at: https://developers.cloudflare.com/ai-search/llms.txt > Use this file to discover all available pages before exploring further. [Skip to content](#%5Ftop) # Configuration You can customize how your AI Search instance indexes your data, retrieves results, and generates responses. Some settings can be updated after the instance is created, while others are fixed at creation time. ## Data source | Configuration | Editable after creation | Description | | ----------------------------------------------------------------------------------------------------------- | ----------------------- | -------------------------------------------------------- | | [Built-in storage](https://developers.cloudflare.com/ai-search/configuration/data-source/built-in-storage/) | n/a | Upload files directly to an instance | | [Website](https://developers.cloudflare.com/ai-search/configuration/data-source/website/) | no | Connect a domain you own to index website pages | | [R2 Bucket](https://developers.cloudflare.com/ai-search/configuration/data-source/r2/) | no | Connect a Cloudflare R2 bucket to index stored documents | ## Indexing | Configuration | Editable after creation | Description | | ---------------------------------------------------------------------------------------------------------- | ----------------------- | --------------------------------------------------------------- | | [Vector search](https://developers.cloudflare.com/ai-search/configuration/indexing/vector-search/) | yes | Vector search and the built-in vector index | | [Path filtering](https://developers.cloudflare.com/ai-search/configuration/indexing/path-filtering/) | yes | Include or exclude specific paths from indexing | | [Chunking](https://developers.cloudflare.com/ai-search/configuration/indexing/chunking/) | yes | Number of tokens per chunk and overlap between chunks | | [Syncing](https://developers.cloudflare.com/ai-search/configuration/indexing/syncing/) | yes | Sync jobs and indexing controls | | [Keyword search](https://developers.cloudflare.com/ai-search/configuration/indexing/keyword-search/) | yes | Enable keyword (BM25) search for exact term matching | | [Hybrid search](https://developers.cloudflare.com/ai-search/configuration/indexing/hybrid-search/) | yes | Combine vector and keyword search with configurable fusion | | [Metadata](https://developers.cloudflare.com/ai-search/configuration/indexing/metadata/) | yes | Custom metadata fields for filtering | | [Service API token](https://developers.cloudflare.com/ai-search/configuration/indexing/service-api-token/) | yes | API token that grants AI Search permission to access R2 buckets | ## Retrieval | Configuration | Editable after creation | Description | | --------------------------------------------------------------------------------------------------------- | ----------------------- | ------------------------------------------------------------- | | [Result filtering](https://developers.cloudflare.com/ai-search/configuration/retrieval/result-filtering/) | yes | Match threshold and maximum number of results | | [Relevance boosting](https://developers.cloudflare.com/ai-search/configuration/retrieval/boosting/) | yes | Bias results by metadata characteristics | | [Reranking](https://developers.cloudflare.com/ai-search/configuration/retrieval/reranking/) | yes | Reorder results by semantic relevance using a reranking model | | [Query rewriting](https://developers.cloudflare.com/ai-search/configuration/retrieval/query-rewriting/) | yes | Rewrite follow-up queries using conversation context | | [System prompt](https://developers.cloudflare.com/ai-search/configuration/retrieval/system-prompt/) | yes | Guide query rewriting and response generation behavior | | [Similarity caching](https://developers.cloudflare.com/ai-search/configuration/retrieval/cache/) | yes | Cache responses for similar prompts | | [Public endpoint](https://developers.cloudflare.com/ai-search/configuration/retrieval/public-endpoint/) | yes | Enable public access to search, chat, and MCP endpoints | | [UI snippets](https://developers.cloudflare.com/ai-search/configuration/retrieval/embed-search-snippets/) | yes | Embed pre-built search and chat components in your website | ## Models | Configuration | Editable after creation | Description | | ------------------------------------------------------------------------------------------ | ----------------------- | --------------------------------------------------- | | [Embedding model](https://developers.cloudflare.com/ai-search/configuration/models/) | no | Model used to generate vector embeddings | | [Generation model](https://developers.cloudflare.com/ai-search/configuration/models/) | yes | Model used to generate the final response | | [Query rewriting model](https://developers.cloudflare.com/ai-search/configuration/models/) | yes | Model used for query rewriting | | [Reranking model](https://developers.cloudflare.com/ai-search/configuration/models/) | yes | Model used to reorder results by semantic relevance | ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/ai-search/","name":"AI Search"}},{"@type":"ListItem","position":3,"item":{"@id":"/ai-search/configuration/","name":"Configuration"}}]} ``` --- --- title: Data source description: Connect a website, R2 bucket, or upload files directly to your AI Search instance for indexing. image: https://developers.cloudflare.com/dev-products-preview.png --- > Documentation Index > Fetch the complete documentation index at: https://developers.cloudflare.com/ai-search/llms.txt > Use this file to discover all available pages before exploring further. [Skip to content](#%5Ftop) # Data source You can upload files directly to an instance or connect an external data source. | Data Source | Description | | ----------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | | [Built-in storage](https://developers.cloudflare.com/ai-search/configuration/data-source/built-in-storage/) | Upload files directly to an instance. Available by default on every new instance. | | [Website](https://developers.cloudflare.com/ai-search/configuration/data-source/website/) | Connect a domain you own to index website pages. | | [R2 Bucket](https://developers.cloudflare.com/ai-search/configuration/data-source/r2/) | Connect a Cloudflare R2 bucket to index stored documents. | ## Supported file types AI Search can ingest a variety of file types. The following plain text files and rich format files are supported. ### Plain text file types | Format | File extensions | Mime Type | | ---------- | ---------------------------------------------------------------- | -------------------------------------------------------------- | | Text | .txt, .rst | text/plain | | Log | .log | text/plain | | Config | .ini, .conf, .env, .properties, .gitignore, .editorconfig, .toml | text/plain, text/toml | | Markdown | .markdown, .md, .mdx | text/markdown | | LaTeX | .tex, .latex | application/x-tex, application/x-latex | | Script | .sh, .bat, .ps1 | application/x-sh, application/x-msdos-batch, text/x-powershell | | SGML | .sgml | text/sgml | | JSON | .json | application/json | | YAML | .yaml, .yml | application/x-yaml | | CSS | .css | text/css | | JavaScript | .js | application/javascript | | PHP | .php | application/x-httpd-php | | Python | .py | text/x-python | | Ruby | .rb | text/x-ruby | | Java | .java | text/x-java-source | | C | .c | text/x-c | | C++ | .cpp, .cxx | text/x-c++ | | C Header | .h, .hpp | text/x-c-header | | Go | .go | text/x-go | | Rust | .rs | text/rust | | Swift | .swift | text/swift | | Dart | .dart | text/dart | | EMACS Lisp | .el | application/x-elisp, text/x-elisp, text/x-emacs-lisp | ### Rich format file types AI Search uses [Markdown Conversion](https://developers.cloudflare.com/workers-ai/features/markdown-conversion/) to convert rich format files to markdown. The following table lists the supported formats that will be converted to Markdown: | Format | File extensions | Mime Types | | PDF Documents | .pdf | application/pdf | | -------------------------- | ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | ---- | --------------- | | Images 1 | .jpeg, .jpg, .png, .webp, .svg | image/jpeg, image/png, image/webp, image/svg+xml | | | | | HTML Documents | .html, .htm | text/html | | | | | XML Documents | .xml | application/xml | | | | | Microsoft Office Documents | .xlsx, .xlsm, .xlsb, .xls, .et, .docx | application/vnd.openxmlformats-officedocument.spreadsheetml.sheet,application/vnd.ms-excel.sheet.macroenabled.12,application/vnd.ms-excel.sheet.binary.macroenabled.12,application/vnd.ms-excel,application/vnd.openxmlformats-officedocument.wordprocessingml.document | | | | | Open Document Format | .ods, .odt | application/vnd.oasis.opendocument.spreadsheet,application/vnd.oasis.opendocument.text | | | | | CSV | .csv | text/csv | | | | | Apple Documents | .numbers | application/vnd.apple.numbers | | | | 1 Image conversion uses two Workers AI models for object detection and summarization. See [Workers AI pricing](https://developers.cloudflare.com/workers-ai/features/markdown-conversion/#pricing) for more details. ## File limits AI Search has a file size limit of **up to 4 MB**. Files that exceed this limit will not be indexed and will show up in the error logs. ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/ai-search/","name":"AI Search"}},{"@type":"ListItem","position":3,"item":{"@id":"/ai-search/configuration/","name":"Configuration"}},{"@type":"ListItem","position":4,"item":{"@id":"/ai-search/configuration/data-source/","name":"Data source"}}]} ``` --- --- title: Built-in storage description: Upload files directly to an AI Search instance using built-in storage powered by R2 and Vectorize. image: https://developers.cloudflare.com/dev-products-preview.png --- > Documentation Index > Fetch the complete documentation index at: https://developers.cloudflare.com/ai-search/llms.txt > Use this file to discover all available pages before exploring further. [Skip to content](#%5Ftop) # Built-in storage Every new AI Search instance comes with built-in storage and a built-in vector index, powered by [R2](https://developers.cloudflare.com/r2/) and [Vectorize](https://developers.cloudflare.com/vectorize/). You can upload files directly to an instance without setting up either service yourself. ## Instances created after April 16, 2026 Instances created after **April 16, 2026** include built-in storage. You can upload files directly using the [Items API](https://developers.cloudflare.com/ai-search/api/items/workers-binding/) or the dashboard. Instances created before this date do not include built-in storage and do not support the Items API. You can see which of your instances have built-in storage on the dashboard under **AI** \> **AI Search**. Select your instance and check the **Overview** tab to see if it includes built-in storage. [ Go to **AI Search** ](https://dash.cloudflare.com/?to=/:account/ai/ai-search) All instances created before this date will be automatically migrated to managed infrastructure on **June 3, 2026**. For details, refer to [Limits & pricing](https://developers.cloudflare.com/ai-search/platform/limits-pricing/#previous-instances). ## Upload and manage files Upload files to an instance using the [Items API](https://developers.cloudflare.com/ai-search/api/items/workers-binding/) (Workers binding or REST API) or the **Items** tab in the dashboard (**AI** \> **AI Search** \> your instance > **Items**). You can also list, view, and delete uploaded files through the Items API or the dashboard. For supported file types, refer to [Supported file types](https://developers.cloudflare.com/ai-search/configuration/data-source/#supported-file-types). ## Indexing Files uploaded to built-in storage are indexed immediately. External data sources like websites and R2 buckets are indexed on a sync schedule. ## External data sources An instance can use built-in storage alongside an external data source. The available external data sources are: * [Website](https://developers.cloudflare.com/ai-search/configuration/data-source/website/): crawl and index a website that you own * [R2 Bucket](https://developers.cloudflare.com/ai-search/configuration/data-source/r2/): index documents stored in a Cloudflare R2 bucket For example, an instance can be backed by a website for shared documentation while also accepting file uploads through the Items API for additional content. ## Limits and pricing For instances with built-in storage, R2 storage, Vectorize, and Browser Run are included. Workers AI and AI Gateway usage is billed separately. For full details, refer to [Limits and pricing](https://developers.cloudflare.com/ai-search/platform/limits-pricing/). ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/ai-search/","name":"AI Search"}},{"@type":"ListItem","position":3,"item":{"@id":"/ai-search/configuration/","name":"Configuration"}},{"@type":"ListItem","position":4,"item":{"@id":"/ai-search/configuration/data-source/","name":"Data source"}},{"@type":"ListItem","position":5,"item":{"@id":"/ai-search/configuration/data-source/built-in-storage/","name":"Built-in storage"}}]} ``` --- --- title: R2 description: Connect a Cloudflare R2 bucket as a data source to index stored documents with AI Search. image: https://developers.cloudflare.com/dev-products-preview.png --- > Documentation Index > Fetch the complete documentation index at: https://developers.cloudflare.com/ai-search/llms.txt > Use this file to discover all available pages before exploring further. [Skip to content](#%5Ftop) # R2 You can connect a [Cloudflare R2](https://developers.cloudflare.com/r2/) bucket as a data source for your AI Search instance. AI Search indexes the files stored in your bucket automatically. ## Get started You can connect an R2 bucket when creating a new instance through the [dashboard](https://developers.cloudflare.com/ai-search/get-started/dashboard/), the [REST API](https://developers.cloudflare.com/ai-search/get-started/api/), or [Wrangler](https://developers.cloudflare.com/ai-search/get-started/wrangler/). R2 is an optional data source that you can add alongside [built-in storage](https://developers.cloudflare.com/ai-search/configuration/data-source/built-in-storage/). If you have never created an R2-backed instance before, we recommend using the dashboard or Wrangler CLI, which will create and register a [service API token](https://developers.cloudflare.com/ai-search/configuration/indexing/service-api-token/) for you automatically. If you are using the REST API or Workers binding, you will need to create a service API token and pass the `token_id` in the create request. Refer to [Service API token](https://developers.cloudflare.com/ai-search/configuration/indexing/service-api-token/) for setup instructions. To get started, [configure an R2 bucket](https://developers.cloudflare.com/r2/get-started/) containing your data. Files that are unsupported or exceed the size limit will be skipped during indexing and logged as errors. ## Path filtering You can control which files get indexed by defining include and exclude rules for object paths. Use this to limit indexing to specific folders or to exclude files you do not want searchable. For example, to index only documentation while excluding drafts: * **Include:** `/docs/**` * **Exclude:** `/docs/drafts/**` Refer to [Path filtering](https://developers.cloudflare.com/ai-search/configuration/indexing/path-filtering/) for pattern syntax, filtering behavior, and more examples. For supported file types and size limits, refer to [Data source](https://developers.cloudflare.com/ai-search/configuration/data-source/#supported-file-types). ## Custom metadata You can attach custom metadata to R2 objects for filtering search results. AI Search reads metadata from S3-compatible custom headers (`x-amz-meta-*`). Before metadata can be extracted, you must [define a schema](https://developers.cloudflare.com/ai-search/configuration/indexing/metadata/#define-a-schema) in your AI Search configuration. ### Set metadata when uploading * [ Workers R2 binding ](#tab-panel-6474) * [ AWS SDK ](#tab-panel-6475) * [ Wrangler CLI ](#tab-panel-6476) Use the `customMetadata` option when uploading objects with the [R2 Workers binding](https://developers.cloudflare.com/r2/api/workers/workers-api-usage/): JavaScript ``` await env.MY_BUCKET.put("docs/document.pdf", fileContent, { customMetadata: { category: "documentation", version: "2.5", is_public: "true", }, }); ``` Use the `Metadata` option with the [AWS SDK for JavaScript](https://developers.cloudflare.com/r2/api/s3/api/): JavaScript ``` import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3"; const client = new S3Client({ region: "auto", endpoint: `https://${accountId}.r2.cloudflarestorage.com`, credentials: { accessKeyId: R2_ACCESS_KEY_ID, secretAccessKey: R2_SECRET_ACCESS_KEY, }, }); await client.send( new PutObjectCommand({ Bucket: "your-bucket", Key: "docs/document.pdf", Body: fileContent, Metadata: { category: "documentation", version: "2.5", is_public: "true", }, }), ); ``` Use the `--header` flag with [Wrangler](https://developers.cloudflare.com/r2/reference/wrangler-commands/) to set `x-amz-meta-*` headers: Terminal window ``` wrangler r2 object put your-bucket/docs/document.pdf \ --file=./document.pdf \ --header="x-amz-meta-category:documentation" \ --header="x-amz-meta-version:2.5" \ --header="x-amz-meta-is_public:true" ``` ### How metadata extraction works When a file is fetched from R2 during indexing: 1. All `x-amz-meta-*` headers are read from the object. 2. The `x-amz-meta-` prefix is stripped (for example, `x-amz-meta-category` becomes `category`). 3. Field names are matched against your schema (case-insensitive). 4. Values are cast to the configured data type. 5. Invalid values (for example, a non-numeric string for a `number` type) are silently ignored. ### Unicode support Metadata values support Unicode characters through MIME-Word encoding (RFC 2047). Most S3-compatible tools handle this encoding automatically. ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/ai-search/","name":"AI Search"}},{"@type":"ListItem","position":3,"item":{"@id":"/ai-search/configuration/","name":"Configuration"}},{"@type":"ListItem","position":4,"item":{"@id":"/ai-search/configuration/data-source/","name":"Data source"}},{"@type":"ListItem","position":5,"item":{"@id":"/ai-search/configuration/data-source/r2/","name":"R2"}}]} ``` --- --- title: Website description: Connect a domain you own as a data source so AI Search can crawl and index your website pages. image: https://developers.cloudflare.com/dev-products-preview.png --- > Documentation Index > Fetch the complete documentation index at: https://developers.cloudflare.com/ai-search/llms.txt > Use this file to discover all available pages before exploring further. [Skip to content](#%5Ftop) # Website You can connect a website you own as a data source for your AI Search instance. AI Search crawls and indexes the pages automatically. You can only crawl domains that you have onboarded onto the same Cloudflare account. Refer to [Onboard a domain](https://developers.cloudflare.com/fundamentals/manage-domains/add-site/) for more information on adding a domain to your Cloudflare account. Bot protection may block crawling If you use Cloudflare products that control or restrict bot traffic such as [Bot Management](https://developers.cloudflare.com/bots/), [Web Application Firewall (WAF)](https://developers.cloudflare.com/waf/), or [Turnstile](https://developers.cloudflare.com/turnstile/), the same rules will apply to the AI Search crawler. Make sure to configure an exception or an allow-list for the AI Search crawler in your settings. ## Get started You can connect a website when creating a new instance through the [dashboard](https://developers.cloudflare.com/ai-search/get-started/dashboard/), the [REST API](https://developers.cloudflare.com/ai-search/get-started/api/), or [Wrangler](https://developers.cloudflare.com/ai-search/get-started/wrangler/). Website is an optional data source that you can add alongside [built-in storage](https://developers.cloudflare.com/ai-search/configuration/data-source/built-in-storage/). ## How website crawling works When you connect a domain, the crawler looks for your website's sitemap to determine which pages to visit: 1. If you configure one or more custom sitemap URLs in the dashboard under **Parser options** \> **Specific sitemap**, AI Search crawls only those sitemap URLs. 2. Otherwise, the crawler checks `robots.txt` for listed sitemaps. 3. If no `robots.txt` is found, the crawler checks for a sitemap at `/sitemap.xml`. 4. If no sitemap is available, the domain cannot be crawled. ### Indexing order If your sitemaps include `` attributes, AI Search reads all sitemaps and indexes pages based on each page's priority value, regardless of which sitemap the page is in. If no `` is specified, pages are indexed in the order the sitemaps are provided, either from the configured custom sitemap URLs or from `robots.txt` from top to bottom. AI Search supports `.gz` compressed sitemaps. Both `robots.txt` and sitemaps can use partial URLs. ### Sync and updates During scheduled or manual [sync jobs](https://developers.cloudflare.com/ai-search/configuration/indexing/syncing/), the crawler will check for changes to the `` attribute in your sitemap. If it has been changed to a date occurring after the last sync date, then the page is crawled, the updated version is stored, and the page is automatically reindexed so that your search results always reflect the latest content. If the `` attribute is not defined, AI Search uses the `` attribute to determine how often to re-crawl the URL. If neither `` nor `` is defined, AI Search automatically crawls each link once a day. ## Storage For instances with [built-in storage](https://developers.cloudflare.com/ai-search/configuration/data-source/built-in-storage/), crawled pages are stored in managed storage automatically. For older instances created before **April 16, 2026**, AI Search creates a dedicated R2 bucket in your account to store crawled pages. This bucket is automatically managed and is used only for content discovered by the crawler. Note For instances with a dedicated R2 bucket, do not modify the bucket directly as it may disrupt the indexing flow and cause content to not be updated properly. ## Configuration ### Path filtering You can control which pages get indexed by defining include and exclude rules for URL paths. Use this to limit indexing to specific sections of your site or to exclude content you do not want searchable. Note Path filtering matches against the full URL, including the scheme, hostname, and subdomains. For example, a page at `https://www.example.com/blog/post` requires a pattern like `**/blog/**` to match. Using `/blog/**` alone will not match because it does not account for the hostname. For example, to index only blog posts while excluding drafts: * **Include:** `**/blog/**` * **Exclude:** `**/blog/drafts/**` Refer to [Path filtering](https://developers.cloudflare.com/ai-search/configuration/indexing/path-filtering/) for pattern syntax, filtering behavior, and more examples. For supported file types and size limits, refer to [Data source](https://developers.cloudflare.com/ai-search/configuration/data-source/#supported-file-types). ### Parsing options You can configure parsing options during onboarding or in your instance settings under **Parser options**. #### Specific sitemap By default, AI Search crawls all sitemaps listed in your `robots.txt` in the order they appear (top to bottom). If you do not want the crawler to index everything, or if your sitemap is hosted at a non-standard path, you can configure custom sitemap URLs in the dashboard under **Parser options** \> **Specific sitemap**. When custom sitemap URLs are configured, AI Search uses those sitemap URLs instead of auto-discovering sitemaps from `robots.txt` or `/sitemap.xml`. You can add up to five sitemap URLs. #### Rendering mode You can choose how pages are parsed during crawling: * **Static sites**: Downloads the raw HTML for each page. * **Rendered sites**: Loads pages with a headless browser and downloads the fully rendered version, including dynamic JavaScript content. For instances with [built-in storage](https://developers.cloudflare.com/ai-search/configuration/data-source/built-in-storage/), Browser Run is included. For older instances, [Browser Run](https://developers.cloudflare.com/browser-run/pricing/) limits and billing apply. ### Extra headers If your website has pages behind authentication or pages that are only visible to logged-in users, you can configure custom HTTP headers to allow the AI Search crawler to access this protected content. You can add up to five custom HTTP headers to the requests AI Search sends when crawling your site. #### Providing access to sites protected by Cloudflare Access To allow AI Search to crawl a site protected by [Cloudflare Access](https://developers.cloudflare.com/cloudflare-one/access-controls/), you need to create service token credentials and configure them as custom headers. Service tokens bypass user authentication, so ensure your Access policies are configured appropriately for the content you want to index. The service token will allow the AI Search crawler to access all content covered by the Service Auth policy. 1. In the [Cloudflare dashboard ↗](https://dash.cloudflare.com/), [create a service token](https://developers.cloudflare.com/cloudflare-one/access-controls/service-credentials/service-tokens/#create-a-service-token). Once the Client ID and Client Secret are generated, save them for the next steps. For example they can look like: ``` CF-Access-Client-Id: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.access CF-Access-Client-Secret: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ``` 2. [Create a policy](https://developers.cloudflare.com/cloudflare-one/access-controls/policies/policy-management/#create-a-policy) with the following configuration: * Add an **Include** rule with **Selector** set to **Service token**. * In **Value**, select the Service Token you created in step 1. 3. [Add your self-hosted application to Access](https://developers.cloudflare.com/cloudflare-one/access-controls/applications/http-apps/self-hosted-public-app/) with the following configuration: * In Access policies, click **Select existing policies**. * Select the policy that you have just created and select **Confirm**. 4. In the Cloudflare dashboard, go to the **AI Search** page. [ Go to **AI Search** ](https://dash.cloudflare.com/?to=/:account/ai/ai-search) 5. Select **Create**. 6. Select **Website** as your data source. 7. Under **Parse options**, locate **Extra headers** and add the following two headers using your saved credentials: * Header 1: * **Key**: `CF-Access-Client-Id` * **Value**: `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.access` * Header 2: * **Key**: `CF-Access-Client-Secret` * **Value**: `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx` 8. Complete the AI Search setup process to create your search instance. ## Custom metadata You can attach custom metadata to web pages using HTML `` tags. AI Search extracts metadata from the `` section of each crawled page. Before custom metadata can be extracted, you must [define a schema](https://developers.cloudflare.com/ai-search/configuration/indexing/metadata/#define-a-schema) in your AI Search configuration. ### Add metadata to web pages Add `` tags using either the `name` or `property` attribute: ``` ``` ### Recognized fields For the following fields, AI Search knows which meta tags to extract from. You must still define these in your schema to enable extraction. | Field | Source | | ----------- | ------------------------------------------------------------- | | title | or | | description | or | | image | | When both a standard meta tag and an Open Graph tag are present, the standard meta tag takes precedence. ### How metadata extraction works When the crawler fetches a page: 1. All `` tags with `name` or `property` attributes are parsed from the `` section. 2. Tag names are matched against your schema (case-insensitive). 3. The `content` attribute value is cast to the configured data type. 4. Extracted metadata is stored alongside the cached HTML. 5. On subsequent processing, metadata flows into the vector index. ### Boolean value parsing For `boolean` fields, the following values are accepted (case-insensitive): | True values | False values | | ------------ | ------------ | | true, 1, yes | false, 0, no | Any other value is treated as invalid and the field is omitted. ## Content selectors Content selectors let you control which parts of a crawled page are indexed. Each entry pairs a URL glob pattern with a CSS selector. When a page URL matches a glob pattern, only the elements matching the corresponding CSS selector — and their descendants — are extracted and converted to Markdown for indexing. The list is ordered and the **first matching path wins**. If a page URL matches multiple glob patterns, only the selector from the first match is applied. Order your entries from most specific to least specific. ### Default behavior Without content selectors, AI Search applies a default processing pipeline that removes elements such as `
`, `