AI Search makes it easier to build a search engine for your data
AI Search gets you from a data source to a working search endpoint quickly. This release adds what you need to put that endpoint in front of real users: your own domain, authentication, and one endpoint across several instances. It also adds crawling for sites without a complete sitemap, so your index covers everything you want it to find.
Each of the following is a new option. The previous behavior is still the default, so nothing changes until you change it.
A public endpoint is a URL that a site or app can query directly, with no authentication in front of it. By default that URL is a generated hostname on search.ai.cloudflare.com. You can now serve the same endpoint from a custom domain, a hostname in a zone that you own:
https://search.example.com/searchOnce your endpoint is on your own domain, you can put Cloudflare Access in front of it. For example, you usually want to give /mcp to specific agents rather than to anyone who finds the URL. Agents authenticate with an Access service token, and people who open the endpoint in a browser sign in through your identity provider.
A namespace can expose its own public endpoint with /search, /chat/completions, and /mcp paths that fan out across the instances you choose:
curl https://ns-<NAMESPACE_ENDPOINT_ID>.search.ai.cloudflare.com/search \
--header "Content-Type: application/json" \
--data '{
"messages": [{ "content": "How do I configure AI Search?", "role": "user" }],
"ai_search_options": { "instance_ids": ["docs", "support"] }
}'Website data sources support a new discover parse type. It starts at the source URL and collects pages from both your sitemaps and the links it finds while crawling:
curl -X POST "https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/ai-search/instances" \
-H "Authorization: Bearer <API_TOKEN>" \
-H "Content-Type: application/json" \
-d '{
"id": "my-ai-search",
"type": "web-crawler",
"source": "example.com",
"source_params": {
"web_crawler": {
"parse_type": "discover",
"discover_options": { "source": "links", "limit": 5000, "depth": 3 }
}
}
}'To learn more, refer to the AI Search documentation.