Skip to content
Cloudflare API
Start here
API Reference
Radar
Robots TXT
Top
User Agents

Get top user agents on robots.txt files

GET/radar/robots_txt/top/user_agents/directive

Retrieves the top user agents on robots.txt files.

Security
API Token

The preferred authorization scheme for interacting with the Cloudflare API. Create a token.

Example:Authorization: Bearer Sn3lZJTBX6kkg7OdcBUAxOO963GEIyGQqnFTOFYY
API Email + API Key

The previous authorization scheme for interacting with the Cloudflare API, used in conjunction with a Global API key.

Example:X-Auth-Email: user@example.com

The previous authorization scheme for interacting with the Cloudflare API. When possible, use API tokens instead of Global API keys.

Example:X-Auth-Key: 144c9defac04969c7bfad8efaa8ea194
Accepted Permissions (at least one required)
User Details WriteUser Details Read
Query ParametersExpand Collapse
date: optional array of string

Filters results by the specified array of dates.

directive: optional "ALLOW" or "DISALLOW"

Filters results by robots.txt directive.

One of the following:
"ALLOW"
"DISALLOW"
domainCategory: optional array of string

Filters results by domain category.

format: optional "JSON" or "CSV"

Format in which results will be returned.

One of the following:
"JSON"
"CSV"
limit: optional number

Limits the number of objects returned in the response.

exclusiveMinimum
minimum0
name: optional array of string

Array of names used to label the series in the response.

userAgentCategory: optional "AI"

Filters results by user agent category.

ReturnsExpand Collapse
result: { meta, top_0 }
meta: { confidenceInfo, dateRange, lastUpdated, 2 more }

Metadata for the results.

confidenceInfo: { annotations, level }
annotations: array of { dataSource, description, endDate, 4 more }
dataSource: "ALL" or "AI_BOTS" or "AI_GATEWAY" or 22 more

Data source for annotations.

One of the following:
"ALL"
"AI_BOTS"
"AI_GATEWAY"
"BGP"
"BOTS"
"CONNECTION_ANOMALY"
"CT"
"DNS"
"DNS_MAGNITUDE"
"DNS_AS112"
"DOS"
"EMAIL_ROUTING"
"EMAIL_SECURITY"
"FW"
"FW_PG"
"HTTP"
"HTTP_CONTROL"
"HTTP_CRAWLER_REFERER"
"HTTP_ORIGINS"
"IQI"
"LEAKED_CREDENTIALS"
"NET"
"ROBOTS_TXT"
"SPEED"
"WORKERS_AI"
description: string
endDate: string
formatdate-time
eventType: "EVENT" or "GENERAL" or "OUTAGE" or 3 more

Event type for annotations.

One of the following:
"EVENT"
"GENERAL"
"OUTAGE"
"PARTIAL_PROJECTION"
"PIPELINE"
"TRAFFIC_ANOMALY"
isInstantaneous: boolean

Whether event is a single point in time or a time range.

linkedUrl: string
formaturi
startDate: string
formatdate-time
level: number

Provides an indication of how much confidence Cloudflare has in the data.

dateRange: array of { endTime, startTime }
endTime: string

Adjusted end of date range.

formatdate-time
startTime: string

Adjusted start of date range.

formatdate-time
lastUpdated: string

Timestamp of the last dataset update.

formatdate-time
normalization: "PERCENTAGE" or "MIN0_MAX" or "MIN_MAX" or 5 more

Normalization method applied to the results. Refer to Normalization methods.

One of the following:
"PERCENTAGE"
"MIN0_MAX"
"MIN_MAX"
"RAW_VALUES"
"PERCENTAGE_CHANGE"
"ROLLING_AVERAGE"
"OVERLAPPED_PERCENTAGE"
"RATIO"
units: array of { name, value }

Measurement units for the results.

name: string
value: string
top_0: array of { name, value, fully, partially }
name: string
value: number
fully: optional number
partially: optional number
success: boolean

Get top user agents on robots.txt files

curl https://api.cloudflare.com/client/v4/radar/robots_txt/top/user_agents/directive \
    -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN"
{
  "result": {
    "meta": {
      "confidenceInfo": {
        "annotations": [
          {
            "dataSource": "ALL",
            "description": "Cable cut in Tonga",
            "endDate": "2019-12-27T18:11:19.117Z",
            "eventType": "EVENT",
            "isInstantaneous": true,
            "linkedUrl": "https://example.com",
            "startDate": "2019-12-27T18:11:19.117Z"
          }
        ],
        "level": 0
      },
      "dateRange": [
        {
          "endTime": "2022-09-17T10:22:57.555Z",
          "startTime": "2022-09-16T10:22:57.555Z"
        }
      ],
      "lastUpdated": "2019-12-27T18:11:19.117Z",
      "normalization": "PERCENTAGE",
      "units": [
        {
          "name": "*",
          "value": "requests"
        }
      ]
    },
    "top_0": [
      {
        "name": "GPTBot",
        "value": 273,
        "fully": 235,
        "partially": 38
      }
    ]
  },
  "success": true
}
Returns Examples
{
  "result": {
    "meta": {
      "confidenceInfo": {
        "annotations": [
          {
            "dataSource": "ALL",
            "description": "Cable cut in Tonga",
            "endDate": "2019-12-27T18:11:19.117Z",
            "eventType": "EVENT",
            "isInstantaneous": true,
            "linkedUrl": "https://example.com",
            "startDate": "2019-12-27T18:11:19.117Z"
          }
        ],
        "level": 0
      },
      "dateRange": [
        {
          "endTime": "2022-09-17T10:22:57.555Z",
          "startTime": "2022-09-16T10:22:57.555Z"
        }
      ],
      "lastUpdated": "2019-12-27T18:11:19.117Z",
      "normalization": "PERCENTAGE",
      "units": [
        {
          "name": "*",
          "value": "requests"
        }
      ]
    },
    "top_0": [
      {
        "name": "GPTBot",
        "value": 273,
        "fully": 235,
        "partially": 38
      }
    ]
  },
  "success": true
}