Skip to content
Cloudflare API
Start here
API Reference
AI Audit
Robots

Get robots.txt rules

client.aiAudit.robots.get(RobotGetParams { zone_id, subdomain } params, RequestOptionsoptions?): RobotGetResponse { userAgents, sitemaps, status }
GET/zones/{zone_id}/ai-audit/robots

Fetches and parses the robots.txt file for a zone or a specific subdomain within the zone. Returns parsed user-agent rules, content signals, and sitemaps.

Security
API Token

The preferred authorization scheme for interacting with the Cloudflare API. Create a token.

Example:Authorization: Bearer Sn3lZJTBX6kkg7OdcBUAxOO963GEIyGQqnFTOFYY
API Email + API Key

The previous authorization scheme for interacting with the Cloudflare API, used in conjunction with a Global API key.

Example:X-Auth-Email: user@example.com

The previous authorization scheme for interacting with the Cloudflare API. When possible, use API tokens instead of Global API keys.

Example:X-Auth-Key: 144c9defac04969c7bfad8efaa8ea194
Accepted Permissions (at least one required)
Zone Settings WriteZone Settings Read
ParametersExpand Collapse
params: RobotGetParams { zone_id, subdomain }
zone_id: string

Path param: Identifier of the zone.

subdomain?: string

Query param: Optional subdomain to fetch robots.txt for. If omitted, fetches robots.txt for the zone apex domain.

ReturnsExpand Collapse
RobotGetResponse { userAgents, sitemaps, status }

Parsed robots.txt rules for a single domain.

userAgents: Record<string, UserAgents>

Map of user-agent string to its parsed rules.

allow: Array<string>

List of allowed path patterns.

disallow: Array<string>

List of disallowed path patterns.

contentSignals?: ContentSignals { ai-input, ai-train, search }

Content signal directives from robots.txt.

"ai-input"?: "yes" | "no"

Whether AI input usage is permitted.

One of the following:
"yes"
"no"
"ai-train"?: "yes" | "no"

Whether AI training is permitted.

One of the following:
"yes"
"no"
One of the following:
crawlDelay?: number

Crawl delay in seconds.

sitemaps?: Array<string>

List of sitemap URLs found in robots.txt.

status?: number

HTTP status code from fetching the robots.txt file.

Get robots.txt rules

import Cloudflare from 'cloudflare';

const client = new Cloudflare({
  apiToken: process.env['CLOUDFLARE_API_TOKEN'], // This is the default and can be omitted
});

const robot = await client.aiAudit.robots.get({ zone_id: 'zone_id' });

console.log(robot.userAgents);
{
  "errors": [],
  "messages": [
    {
      "code": 1000,
      "message": "message",
      "documentation_url": "documentation_url",
      "source": {
        "pointer": "pointer"
      }
    }
  ],
  "success": true,
  "result": {
    "userAgents": {
      "foo": {
        "allow": [
          "string"
        ],
        "disallow": [
          "string"
        ],
        "contentSignals": {
          "ai-input": "yes",
          "ai-train": "yes",
          "search": "yes"
        },
        "crawlDelay": 0
      }
    },
    "sitemaps": [
      "string"
    ],
    "status": 0
  }
}
Returns Examples
{
  "errors": [],
  "messages": [
    {
      "code": 1000,
      "message": "message",
      "documentation_url": "documentation_url",
      "source": {
        "pointer": "pointer"
      }
    }
  ],
  "success": true,
  "result": {
    "userAgents": {
      "foo": {
        "allow": [
          "string"
        ],
        "disallow": [
          "string"
        ],
        "contentSignals": {
          "ai-input": "yes",
          "ai-train": "yes",
          "search": "yes"
        },
        "crawlDelay": 0
      }
    },
    "sitemaps": [
      "string"
    ],
    "status": 0
  }
}