Bulk get robots.txt rules

client.AIAudit.Robots.BulkGet(ctx, params) (*RobotBulkGetResponse, error)

POST/zones/{zone_id}/ai-audit/robots/bulk

Fetches and parses robots.txt files for multiple domains within a zone in a single request. Each domain must belong to the specified zone. Results are keyed by hostname.

Security

API Token

The preferred authorization scheme for interacting with the Cloudflare API. Create a token.

Example:Authorization: Bearer Sn3lZJTBX6kkg7OdcBUAxOO963GEIyGQqnFTOFYY

API Email + API Key

The previous authorization scheme for interacting with the Cloudflare API, used in conjunction with a Global API key.

Example:X-Auth-Email: user@example.com

The previous authorization scheme for interacting with the Cloudflare API. When possible, use API tokens instead of Global API keys.

Example:X-Auth-Key: 144c9defac04969c7bfad8efaa8ea194

Accepted Permissions (at least one required)

Zone Settings WriteZone Settings Read

ParametersExpand Collapse

params RobotBulkGetParams

ZoneID param.Field[string]

Path param: Identifier of the zone.

Body param.Field[[]string]

Body param: Array of domain hostnames to fetch robots.txt for. Each domain must end with the zone name. Maximum 25 domains per request.

ReturnsExpand Collapse

type RobotBulkGetResponse map[string, RobotBulkGetResponseItem]

Map of hostname to parsed robots.txt rules.

UserAgents map[string, RobotBulkGetResponseItemUserAgent]

Map of user-agent string to its parsed rules.

Allow []string

List of allowed path patterns.

Disallow []string

List of disallowed path patterns.

ContentSignals RobotBulkGetResponseItemUserAgentsContentSignalsOptional

Content signal directives from robots.txt.

AIInput RobotBulkGetResponseItemUserAgentsContentSignalsAIInputOptional

Whether AI input usage is permitted.

One of the following:

const RobotBulkGetResponseItemUserAgentsContentSignalsAIInputYes RobotBulkGetResponseItemUserAgentsContentSignalsAIInput = "yes"

const RobotBulkGetResponseItemUserAgentsContentSignalsAIInputNo RobotBulkGetResponseItemUserAgentsContentSignalsAIInput = "no"

AITrain RobotBulkGetResponseItemUserAgentsContentSignalsAITrainOptional

Whether AI training is permitted.

One of the following:

const RobotBulkGetResponseItemUserAgentsContentSignalsAITrainYes RobotBulkGetResponseItemUserAgentsContentSignalsAITrain = "yes"

const RobotBulkGetResponseItemUserAgentsContentSignalsAITrainNo RobotBulkGetResponseItemUserAgentsContentSignalsAITrain = "no"

Search RobotBulkGetResponseItemUserAgentsContentSignalsSearchOptional

Whether search indexing is permitted.

One of the following:

const RobotBulkGetResponseItemUserAgentsContentSignalsSearchYes RobotBulkGetResponseItemUserAgentsContentSignalsSearch = "yes"

const RobotBulkGetResponseItemUserAgentsContentSignalsSearchNo RobotBulkGetResponseItemUserAgentsContentSignalsSearch = "no"

CrawlDelay float64Optional

Crawl delay in seconds.

Sitemaps []stringOptional

List of sitemap URLs found in robots.txt.

Status int64Optional

HTTP status code from fetching the robots.txt file.

Bulk get robots.txt rules

package main

import (
  "context"
  "fmt"

  "github.com/cloudflare/cloudflare-go"
  "github.com/cloudflare/cloudflare-go/ai_audit"
  "github.com/cloudflare/cloudflare-go/option"
)

func main() {
  client := cloudflare.NewClient(
    option.WithAPIToken("Sn3lZJTBX6kkg7OdcBUAxOO963GEIyGQqnFTOFYY"),
  )
  response, err := client.AIAudit.Robots.BulkGet(context.TODO(), ai_audit.RobotBulkGetParams{
    ZoneID: cloudflare.F("zone_id"),
    Body: []string{"example.com", "blog.example.com"},
  })
  if err != nil {
    panic(err.Error())
  }
  fmt.Printf("%+v\n", response)
}

{
  "errors": [],
  "messages": [
    {
      "code": 1000,
      "message": "message",
      "documentation_url": "documentation_url",
      "source": {
        "pointer": "pointer"
      }
    }
  ],
  "success": true,
  "result": {
    "foo": {
      "userAgents": {
        "foo": {
          "allow": [
            "string"
          ],
          "disallow": [
            "string"
          ],
          "contentSignals": {
            "ai-input": "yes",
            "ai-train": "yes",
            "search": "yes"
          },
          "crawlDelay": 0
        }
      },
      "sitemaps": [
        "string"
      ],
      "status": 0
    }
  }
}

Returns Examples

{
  "errors": [],
  "messages": [
    {
      "code": 1000,
      "message": "message",
      "documentation_url": "documentation_url",
      "source": {
        "pointer": "pointer"
      }
    }
  ],
  "success": true,
  "result": {
    "foo": {
      "userAgents": {
        "foo": {
          "allow": [
            "string"
          ],
          "disallow": [
            "string"
          ],
          "contentSignals": {
            "ai-input": "yes",
            "ai-train": "yes",
            "search": "yes"
          },
          "crawlDelay": 0
        }
      },
      "sitemaps": [
        "string"
      ],
      "status": 0
    }
  }
}