Skip to content
Cloudflare API
Start here
API Reference
Radar

Robots TXT

Robots TXTTop

Get top domain categories by robots.txt files parsed
client.radar.robotsTXT.top.domainCategories(TopDomainCategoriesParams { date, format, limit, 2 more } query?, RequestOptionsoptions?): TopDomainCategoriesResponse { meta, top_0 }
GET/radar/robots_txt/top/domain_categories
ModelsExpand Collapse
TopDomainCategoriesResponse { meta, top_0 }
meta: Meta { confidenceInfo, dateRange, lastUpdated, 2 more }

Metadata for the results.

confidenceInfo: ConfidenceInfo | null
annotations: Array<Annotation>
dataSource: "ALL" | "AI_BOTS" | "AI_GATEWAY" | 22 more

Data source for annotations.

One of the following:
"ALL"
"AI_BOTS"
"AI_GATEWAY"
"BGP"
"BOTS"
"CONNECTION_ANOMALY"
"CT"
"DNS"
"DNS_MAGNITUDE"
"DNS_AS112"
"DOS"
"EMAIL_ROUTING"
"EMAIL_SECURITY"
"FW"
"FW_PG"
"HTTP"
"HTTP_CONTROL"
"HTTP_CRAWLER_REFERER"
"HTTP_ORIGINS"
"IQI"
"LEAKED_CREDENTIALS"
"NET"
"ROBOTS_TXT"
"SPEED"
"WORKERS_AI"
description: string
endDate: string
formatdate-time
eventType: "EVENT" | "GENERAL" | "OUTAGE" | 3 more

Event type for annotations.

One of the following:
"EVENT"
"GENERAL"
"OUTAGE"
"PARTIAL_PROJECTION"
"PIPELINE"
"TRAFFIC_ANOMALY"
isInstantaneous: boolean

Whether event is a single point in time or a time range.

linkedUrl: string
formaturi
startDate: string
formatdate-time
level: number

Provides an indication of how much confidence Cloudflare has in the data.

dateRange: Array<DateRange>
endTime: string

Adjusted end of date range.

formatdate-time
startTime: string

Adjusted start of date range.

formatdate-time
lastUpdated: string

Timestamp of the last dataset update.

formatdate-time
normalization: "PERCENTAGE" | "MIN0_MAX" | "MIN_MAX" | 5 more

Normalization method applied to the results. Refer to Normalization methods.

One of the following:
"PERCENTAGE"
"MIN0_MAX"
"MIN_MAX"
"RAW_VALUES"
"PERCENTAGE_CHANGE"
"ROLLING_AVERAGE"
"OVERLAPPED_PERCENTAGE"
"RATIO"
units: Array<Unit>

Measurement units for the results.

name: string
value: string
top_0: Array<Top0>
name: string
value: number

Robots TXTTopUser Agents

Get top user agents on robots.txt files
client.radar.robotsTXT.top.userAgents.directive(UserAgentDirectiveParams { date, directive, domainCategory, 4 more } query?, RequestOptionsoptions?): UserAgentDirectiveResponse { meta, top_0 }
GET/radar/robots_txt/top/user_agents/directive
ModelsExpand Collapse
UserAgentDirectiveResponse { meta, top_0 }
meta: Meta { confidenceInfo, dateRange, lastUpdated, 2 more }

Metadata for the results.

confidenceInfo: ConfidenceInfo | null
annotations: Array<Annotation>
dataSource: "ALL" | "AI_BOTS" | "AI_GATEWAY" | 22 more

Data source for annotations.

One of the following:
"ALL"
"AI_BOTS"
"AI_GATEWAY"
"BGP"
"BOTS"
"CONNECTION_ANOMALY"
"CT"
"DNS"
"DNS_MAGNITUDE"
"DNS_AS112"
"DOS"
"EMAIL_ROUTING"
"EMAIL_SECURITY"
"FW"
"FW_PG"
"HTTP"
"HTTP_CONTROL"
"HTTP_CRAWLER_REFERER"
"HTTP_ORIGINS"
"IQI"
"LEAKED_CREDENTIALS"
"NET"
"ROBOTS_TXT"
"SPEED"
"WORKERS_AI"
description: string
endDate: string
formatdate-time
eventType: "EVENT" | "GENERAL" | "OUTAGE" | 3 more

Event type for annotations.

One of the following:
"EVENT"
"GENERAL"
"OUTAGE"
"PARTIAL_PROJECTION"
"PIPELINE"
"TRAFFIC_ANOMALY"
isInstantaneous: boolean

Whether event is a single point in time or a time range.

linkedUrl: string
formaturi
startDate: string
formatdate-time
level: number

Provides an indication of how much confidence Cloudflare has in the data.

dateRange: Array<DateRange>
endTime: string

Adjusted end of date range.

formatdate-time
startTime: string

Adjusted start of date range.

formatdate-time
lastUpdated: string

Timestamp of the last dataset update.

formatdate-time
normalization: "PERCENTAGE" | "MIN0_MAX" | "MIN_MAX" | 5 more

Normalization method applied to the results. Refer to Normalization methods.

One of the following:
"PERCENTAGE"
"MIN0_MAX"
"MIN_MAX"
"RAW_VALUES"
"PERCENTAGE_CHANGE"
"ROLLING_AVERAGE"
"OVERLAPPED_PERCENTAGE"
"RATIO"
units: Array<Unit>

Measurement units for the results.

name: string
value: string
top_0: Array<Top0>
name: string
value: number
fully?: number
partially?: number