# Robots TXT

# Top

## Get top domain categories by robots.txt files parsed

`client.radar.robotsTXT.top.domainCategories(TopDomainCategoriesParamsquery?, RequestOptionsoptions?): TopDomainCategoriesResponse`

**get** `/radar/robots_txt/top/domain_categories`

Retrieves the top domain categories by the number of robots.txt files parsed.

### Parameters

- `query: TopDomainCategoriesParams`

  - `date?: Array<string>`

    Filters results by the specified array of dates.

  - `format?: "JSON" | "CSV"`

    Format in which results will be returned.

    - `"JSON"`

    - `"CSV"`

  - `limit?: number`

    Limits the number of objects returned in the response.

  - `name?: Array<string>`

    Array of names used to label the series in the response.

  - `userAgentCategory?: "AI"`

    Filters results by user agent category.

    - `"AI"`

### Returns

- `TopDomainCategoriesResponse`

  - `meta: Meta`

    Metadata for the results.

    - `confidenceInfo: ConfidenceInfo | null`

      - `annotations: Array<Annotation>`

        - `dataSource: "ALL" | "AI_BOTS" | "AI_GATEWAY" | 22 more`

          Data source for annotations.

          - `"ALL"`

          - `"AI_BOTS"`

          - `"AI_GATEWAY"`

          - `"BGP"`

          - `"BOTS"`

          - `"CONNECTION_ANOMALY"`

          - `"CT"`

          - `"DNS"`

          - `"DNS_MAGNITUDE"`

          - `"DNS_AS112"`

          - `"DOS"`

          - `"EMAIL_ROUTING"`

          - `"EMAIL_SECURITY"`

          - `"FW"`

          - `"FW_PG"`

          - `"HTTP"`

          - `"HTTP_CONTROL"`

          - `"HTTP_CRAWLER_REFERER"`

          - `"HTTP_ORIGINS"`

          - `"IQI"`

          - `"LEAKED_CREDENTIALS"`

          - `"NET"`

          - `"ROBOTS_TXT"`

          - `"SPEED"`

          - `"WORKERS_AI"`

        - `description: string`

        - `endDate: string`

        - `eventType: "EVENT" | "GENERAL" | "OUTAGE" | 3 more`

          Event type for annotations.

          - `"EVENT"`

          - `"GENERAL"`

          - `"OUTAGE"`

          - `"PARTIAL_PROJECTION"`

          - `"PIPELINE"`

          - `"TRAFFIC_ANOMALY"`

        - `isInstantaneous: boolean`

          Whether event is a single point in time or a time range.

        - `linkedUrl: string`

        - `startDate: string`

      - `level: number`

        Provides an indication of how much confidence Cloudflare has in the data.

    - `dateRange: Array<DateRange>`

      - `endTime: string`

        Adjusted end of date range.

      - `startTime: string`

        Adjusted start of date range.

    - `lastUpdated: string`

      Timestamp of the last dataset update.

    - `normalization: "PERCENTAGE" | "MIN0_MAX" | "MIN_MAX" | 5 more`

      Normalization method applied to the results. Refer to [Normalization methods](https://developers.cloudflare.com/radar/concepts/normalization/).

      - `"PERCENTAGE"`

      - `"MIN0_MAX"`

      - `"MIN_MAX"`

      - `"RAW_VALUES"`

      - `"PERCENTAGE_CHANGE"`

      - `"ROLLING_AVERAGE"`

      - `"OVERLAPPED_PERCENTAGE"`

      - `"RATIO"`

    - `units: Array<Unit>`

      Measurement units for the results.

      - `name: string`

      - `value: string`

  - `top_0: Array<Top0>`

    - `name: string`

    - `value: number`

### Example

```node
import Cloudflare from 'cloudflare';

const client = new Cloudflare({
  apiToken: process.env['CLOUDFLARE_API_TOKEN'], // This is the default and can be omitted
});

const response = await client.radar.robotsTXT.top.domainCategories();

console.log(response.meta);
```

#### Response

```json
{
  "result": {
    "meta": {
      "confidenceInfo": {
        "annotations": [
          {
            "dataSource": "ALL",
            "description": "Cable cut in Tonga",
            "endDate": "2019-12-27T18:11:19.117Z",
            "eventType": "EVENT",
            "isInstantaneous": true,
            "linkedUrl": "https://example.com",
            "startDate": "2019-12-27T18:11:19.117Z"
          }
        ],
        "level": 0
      },
      "dateRange": [
        {
          "endTime": "2022-09-17T10:22:57.555Z",
          "startTime": "2022-09-16T10:22:57.555Z"
        }
      ],
      "lastUpdated": "2019-12-27T18:11:19.117Z",
      "normalization": "PERCENTAGE",
      "units": [
        {
          "name": "*",
          "value": "requests"
        }
      ]
    },
    "top_0": [
      {
        "name": "News & Media",
        "value": 273
      }
    ]
  },
  "success": true
}
```

## Domain Types

### Top Domain Categories Response

- `TopDomainCategoriesResponse`

  - `meta: Meta`

    Metadata for the results.

    - `confidenceInfo: ConfidenceInfo | null`

      - `annotations: Array<Annotation>`

        - `dataSource: "ALL" | "AI_BOTS" | "AI_GATEWAY" | 22 more`

          Data source for annotations.

          - `"ALL"`

          - `"AI_BOTS"`

          - `"AI_GATEWAY"`

          - `"BGP"`

          - `"BOTS"`

          - `"CONNECTION_ANOMALY"`

          - `"CT"`

          - `"DNS"`

          - `"DNS_MAGNITUDE"`

          - `"DNS_AS112"`

          - `"DOS"`

          - `"EMAIL_ROUTING"`

          - `"EMAIL_SECURITY"`

          - `"FW"`

          - `"FW_PG"`

          - `"HTTP"`

          - `"HTTP_CONTROL"`

          - `"HTTP_CRAWLER_REFERER"`

          - `"HTTP_ORIGINS"`

          - `"IQI"`

          - `"LEAKED_CREDENTIALS"`

          - `"NET"`

          - `"ROBOTS_TXT"`

          - `"SPEED"`

          - `"WORKERS_AI"`

        - `description: string`

        - `endDate: string`

        - `eventType: "EVENT" | "GENERAL" | "OUTAGE" | 3 more`

          Event type for annotations.

          - `"EVENT"`

          - `"GENERAL"`

          - `"OUTAGE"`

          - `"PARTIAL_PROJECTION"`

          - `"PIPELINE"`

          - `"TRAFFIC_ANOMALY"`

        - `isInstantaneous: boolean`

          Whether event is a single point in time or a time range.

        - `linkedUrl: string`

        - `startDate: string`

      - `level: number`

        Provides an indication of how much confidence Cloudflare has in the data.

    - `dateRange: Array<DateRange>`

      - `endTime: string`

        Adjusted end of date range.

      - `startTime: string`

        Adjusted start of date range.

    - `lastUpdated: string`

      Timestamp of the last dataset update.

    - `normalization: "PERCENTAGE" | "MIN0_MAX" | "MIN_MAX" | 5 more`

      Normalization method applied to the results. Refer to [Normalization methods](https://developers.cloudflare.com/radar/concepts/normalization/).

      - `"PERCENTAGE"`

      - `"MIN0_MAX"`

      - `"MIN_MAX"`

      - `"RAW_VALUES"`

      - `"PERCENTAGE_CHANGE"`

      - `"ROLLING_AVERAGE"`

      - `"OVERLAPPED_PERCENTAGE"`

      - `"RATIO"`

    - `units: Array<Unit>`

      Measurement units for the results.

      - `name: string`

      - `value: string`

  - `top_0: Array<Top0>`

    - `name: string`

    - `value: number`

# User Agents

## Get top user agents on robots.txt files

`client.radar.robotsTXT.top.userAgents.directive(UserAgentDirectiveParamsquery?, RequestOptionsoptions?): UserAgentDirectiveResponse`

**get** `/radar/robots_txt/top/user_agents/directive`

Retrieves the top user agents on robots.txt files.

### Parameters

- `query: UserAgentDirectiveParams`

  - `date?: Array<string>`

    Filters results by the specified array of dates.

  - `directive?: "ALLOW" | "DISALLOW"`

    Filters results by robots.txt directive.

    - `"ALLOW"`

    - `"DISALLOW"`

  - `domainCategory?: Array<string>`

    Filters results by domain category.

  - `format?: "JSON" | "CSV"`

    Format in which results will be returned.

    - `"JSON"`

    - `"CSV"`

  - `limit?: number`

    Limits the number of objects returned in the response.

  - `name?: Array<string>`

    Array of names used to label the series in the response.

  - `userAgentCategory?: "AI"`

    Filters results by user agent category.

    - `"AI"`

### Returns

- `UserAgentDirectiveResponse`

  - `meta: Meta`

    Metadata for the results.

    - `confidenceInfo: ConfidenceInfo | null`

      - `annotations: Array<Annotation>`

        - `dataSource: "ALL" | "AI_BOTS" | "AI_GATEWAY" | 22 more`

          Data source for annotations.

          - `"ALL"`

          - `"AI_BOTS"`

          - `"AI_GATEWAY"`

          - `"BGP"`

          - `"BOTS"`

          - `"CONNECTION_ANOMALY"`

          - `"CT"`

          - `"DNS"`

          - `"DNS_MAGNITUDE"`

          - `"DNS_AS112"`

          - `"DOS"`

          - `"EMAIL_ROUTING"`

          - `"EMAIL_SECURITY"`

          - `"FW"`

          - `"FW_PG"`

          - `"HTTP"`

          - `"HTTP_CONTROL"`

          - `"HTTP_CRAWLER_REFERER"`

          - `"HTTP_ORIGINS"`

          - `"IQI"`

          - `"LEAKED_CREDENTIALS"`

          - `"NET"`

          - `"ROBOTS_TXT"`

          - `"SPEED"`

          - `"WORKERS_AI"`

        - `description: string`

        - `endDate: string`

        - `eventType: "EVENT" | "GENERAL" | "OUTAGE" | 3 more`

          Event type for annotations.

          - `"EVENT"`

          - `"GENERAL"`

          - `"OUTAGE"`

          - `"PARTIAL_PROJECTION"`

          - `"PIPELINE"`

          - `"TRAFFIC_ANOMALY"`

        - `isInstantaneous: boolean`

          Whether event is a single point in time or a time range.

        - `linkedUrl: string`

        - `startDate: string`

      - `level: number`

        Provides an indication of how much confidence Cloudflare has in the data.

    - `dateRange: Array<DateRange>`

      - `endTime: string`

        Adjusted end of date range.

      - `startTime: string`

        Adjusted start of date range.

    - `lastUpdated: string`

      Timestamp of the last dataset update.

    - `normalization: "PERCENTAGE" | "MIN0_MAX" | "MIN_MAX" | 5 more`

      Normalization method applied to the results. Refer to [Normalization methods](https://developers.cloudflare.com/radar/concepts/normalization/).

      - `"PERCENTAGE"`

      - `"MIN0_MAX"`

      - `"MIN_MAX"`

      - `"RAW_VALUES"`

      - `"PERCENTAGE_CHANGE"`

      - `"ROLLING_AVERAGE"`

      - `"OVERLAPPED_PERCENTAGE"`

      - `"RATIO"`

    - `units: Array<Unit>`

      Measurement units for the results.

      - `name: string`

      - `value: string`

  - `top_0: Array<Top0>`

    - `name: string`

    - `value: number`

    - `fully?: number`

    - `partially?: number`

### Example

```node
import Cloudflare from 'cloudflare';

const client = new Cloudflare({
  apiToken: process.env['CLOUDFLARE_API_TOKEN'], // This is the default and can be omitted
});

const response = await client.radar.robotsTXT.top.userAgents.directive();

console.log(response.meta);
```

#### Response

```json
{
  "result": {
    "meta": {
      "confidenceInfo": {
        "annotations": [
          {
            "dataSource": "ALL",
            "description": "Cable cut in Tonga",
            "endDate": "2019-12-27T18:11:19.117Z",
            "eventType": "EVENT",
            "isInstantaneous": true,
            "linkedUrl": "https://example.com",
            "startDate": "2019-12-27T18:11:19.117Z"
          }
        ],
        "level": 0
      },
      "dateRange": [
        {
          "endTime": "2022-09-17T10:22:57.555Z",
          "startTime": "2022-09-16T10:22:57.555Z"
        }
      ],
      "lastUpdated": "2019-12-27T18:11:19.117Z",
      "normalization": "PERCENTAGE",
      "units": [
        {
          "name": "*",
          "value": "requests"
        }
      ]
    },
    "top_0": [
      {
        "name": "GPTBot",
        "value": 273,
        "fully": 235,
        "partially": 38
      }
    ]
  },
  "success": true
}
```

## Domain Types

### User Agent Directive Response

- `UserAgentDirectiveResponse`

  - `meta: Meta`

    Metadata for the results.

    - `confidenceInfo: ConfidenceInfo | null`

      - `annotations: Array<Annotation>`

        - `dataSource: "ALL" | "AI_BOTS" | "AI_GATEWAY" | 22 more`

          Data source for annotations.

          - `"ALL"`

          - `"AI_BOTS"`

          - `"AI_GATEWAY"`

          - `"BGP"`

          - `"BOTS"`

          - `"CONNECTION_ANOMALY"`

          - `"CT"`

          - `"DNS"`

          - `"DNS_MAGNITUDE"`

          - `"DNS_AS112"`

          - `"DOS"`

          - `"EMAIL_ROUTING"`

          - `"EMAIL_SECURITY"`

          - `"FW"`

          - `"FW_PG"`

          - `"HTTP"`

          - `"HTTP_CONTROL"`

          - `"HTTP_CRAWLER_REFERER"`

          - `"HTTP_ORIGINS"`

          - `"IQI"`

          - `"LEAKED_CREDENTIALS"`

          - `"NET"`

          - `"ROBOTS_TXT"`

          - `"SPEED"`

          - `"WORKERS_AI"`

        - `description: string`

        - `endDate: string`

        - `eventType: "EVENT" | "GENERAL" | "OUTAGE" | 3 more`

          Event type for annotations.

          - `"EVENT"`

          - `"GENERAL"`

          - `"OUTAGE"`

          - `"PARTIAL_PROJECTION"`

          - `"PIPELINE"`

          - `"TRAFFIC_ANOMALY"`

        - `isInstantaneous: boolean`

          Whether event is a single point in time or a time range.

        - `linkedUrl: string`

        - `startDate: string`

      - `level: number`

        Provides an indication of how much confidence Cloudflare has in the data.

    - `dateRange: Array<DateRange>`

      - `endTime: string`

        Adjusted end of date range.

      - `startTime: string`

        Adjusted start of date range.

    - `lastUpdated: string`

      Timestamp of the last dataset update.

    - `normalization: "PERCENTAGE" | "MIN0_MAX" | "MIN_MAX" | 5 more`

      Normalization method applied to the results. Refer to [Normalization methods](https://developers.cloudflare.com/radar/concepts/normalization/).

      - `"PERCENTAGE"`

      - `"MIN0_MAX"`

      - `"MIN_MAX"`

      - `"RAW_VALUES"`

      - `"PERCENTAGE_CHANGE"`

      - `"ROLLING_AVERAGE"`

      - `"OVERLAPPED_PERCENTAGE"`

      - `"RATIO"`

    - `units: Array<Unit>`

      Measurement units for the results.

      - `name: string`

      - `value: string`

  - `top_0: Array<Top0>`

    - `name: string`

    - `value: number`

    - `fully?: number`

    - `partially?: number`