# Robots TXT

# Top

## Get top domain categories by robots.txt files parsed

`radar.robots_txt.top.domain_categories(TopDomainCategoriesParams**kwargs)  -> TopDomainCategoriesResponse`

**get** `/radar/robots_txt/top/domain_categories`

Retrieves the top domain categories by the number of robots.txt files parsed.

### Parameters

- `date: Optional[SequenceNotStr[Union[null, null]]]`

  Filters results by the specified array of dates.

- `format: Optional[Literal["JSON", "CSV"]]`

  Format in which results will be returned.

  - `"JSON"`

  - `"CSV"`

- `limit: Optional[int]`

  Limits the number of objects returned in the response.

- `name: Optional[SequenceNotStr[str]]`

  Array of names used to label the series in the response.

- `user_agent_category: Optional[Literal["AI"]]`

  Filters results by user agent category.

  - `"AI"`

### Returns

- `class TopDomainCategoriesResponse: …`

  - `meta: Meta`

    Metadata for the results.

    - `confidence_info: Optional[MetaConfidenceInfo]`

      - `annotations: List[MetaConfidenceInfoAnnotation]`

        - `data_source: Literal["ALL", "AI_BOTS", "AI_GATEWAY", 22 more]`

          Data source for annotations.

          - `"ALL"`

          - `"AI_BOTS"`

          - `"AI_GATEWAY"`

          - `"BGP"`

          - `"BOTS"`

          - `"CONNECTION_ANOMALY"`

          - `"CT"`

          - `"DNS"`

          - `"DNS_MAGNITUDE"`

          - `"DNS_AS112"`

          - `"DOS"`

          - `"EMAIL_ROUTING"`

          - `"EMAIL_SECURITY"`

          - `"FW"`

          - `"FW_PG"`

          - `"HTTP"`

          - `"HTTP_CONTROL"`

          - `"HTTP_CRAWLER_REFERER"`

          - `"HTTP_ORIGINS"`

          - `"IQI"`

          - `"LEAKED_CREDENTIALS"`

          - `"NET"`

          - `"ROBOTS_TXT"`

          - `"SPEED"`

          - `"WORKERS_AI"`

        - `description: str`

        - `end_date: datetime`

        - `event_type: Literal["EVENT", "GENERAL", "OUTAGE", 3 more]`

          Event type for annotations.

          - `"EVENT"`

          - `"GENERAL"`

          - `"OUTAGE"`

          - `"PARTIAL_PROJECTION"`

          - `"PIPELINE"`

          - `"TRAFFIC_ANOMALY"`

        - `is_instantaneous: bool`

          Whether event is a single point in time or a time range.

        - `linked_url: str`

        - `start_date: datetime`

      - `level: int`

        Provides an indication of how much confidence Cloudflare has in the data.

    - `date_range: List[MetaDateRange]`

      - `end_time: datetime`

        Adjusted end of date range.

      - `start_time: datetime`

        Adjusted start of date range.

    - `last_updated: datetime`

      Timestamp of the last dataset update.

    - `normalization: Literal["PERCENTAGE", "MIN0_MAX", "MIN_MAX", 5 more]`

      Normalization method applied to the results. Refer to [Normalization methods](https://developers.cloudflare.com/radar/concepts/normalization/).

      - `"PERCENTAGE"`

      - `"MIN0_MAX"`

      - `"MIN_MAX"`

      - `"RAW_VALUES"`

      - `"PERCENTAGE_CHANGE"`

      - `"ROLLING_AVERAGE"`

      - `"OVERLAPPED_PERCENTAGE"`

      - `"RATIO"`

    - `units: List[MetaUnit]`

      Measurement units for the results.

      - `name: str`

      - `value: str`

  - `top_0: List[Top0]`

    - `name: str`

    - `value: int`

### Example

```python
import os
from cloudflare import Cloudflare

client = Cloudflare(
    api_token=os.environ.get("CLOUDFLARE_API_TOKEN"),  # This is the default and can be omitted
)
response = client.radar.robots_txt.top.domain_categories()
print(response.meta)
```

#### Response

```json
{
  "result": {
    "meta": {
      "confidenceInfo": {
        "annotations": [
          {
            "dataSource": "ALL",
            "description": "Cable cut in Tonga",
            "endDate": "2019-12-27T18:11:19.117Z",
            "eventType": "EVENT",
            "isInstantaneous": true,
            "linkedUrl": "https://example.com",
            "startDate": "2019-12-27T18:11:19.117Z"
          }
        ],
        "level": 0
      },
      "dateRange": [
        {
          "endTime": "2022-09-17T10:22:57.555Z",
          "startTime": "2022-09-16T10:22:57.555Z"
        }
      ],
      "lastUpdated": "2019-12-27T18:11:19.117Z",
      "normalization": "PERCENTAGE",
      "units": [
        {
          "name": "*",
          "value": "requests"
        }
      ]
    },
    "top_0": [
      {
        "name": "News & Media",
        "value": 273
      }
    ]
  },
  "success": true
}
```

## Domain Types

### Top Domain Categories Response

- `class TopDomainCategoriesResponse: …`

  - `meta: Meta`

    Metadata for the results.

    - `confidence_info: Optional[MetaConfidenceInfo]`

      - `annotations: List[MetaConfidenceInfoAnnotation]`

        - `data_source: Literal["ALL", "AI_BOTS", "AI_GATEWAY", 22 more]`

          Data source for annotations.

          - `"ALL"`

          - `"AI_BOTS"`

          - `"AI_GATEWAY"`

          - `"BGP"`

          - `"BOTS"`

          - `"CONNECTION_ANOMALY"`

          - `"CT"`

          - `"DNS"`

          - `"DNS_MAGNITUDE"`

          - `"DNS_AS112"`

          - `"DOS"`

          - `"EMAIL_ROUTING"`

          - `"EMAIL_SECURITY"`

          - `"FW"`

          - `"FW_PG"`

          - `"HTTP"`

          - `"HTTP_CONTROL"`

          - `"HTTP_CRAWLER_REFERER"`

          - `"HTTP_ORIGINS"`

          - `"IQI"`

          - `"LEAKED_CREDENTIALS"`

          - `"NET"`

          - `"ROBOTS_TXT"`

          - `"SPEED"`

          - `"WORKERS_AI"`

        - `description: str`

        - `end_date: datetime`

        - `event_type: Literal["EVENT", "GENERAL", "OUTAGE", 3 more]`

          Event type for annotations.

          - `"EVENT"`

          - `"GENERAL"`

          - `"OUTAGE"`

          - `"PARTIAL_PROJECTION"`

          - `"PIPELINE"`

          - `"TRAFFIC_ANOMALY"`

        - `is_instantaneous: bool`

          Whether event is a single point in time or a time range.

        - `linked_url: str`

        - `start_date: datetime`

      - `level: int`

        Provides an indication of how much confidence Cloudflare has in the data.

    - `date_range: List[MetaDateRange]`

      - `end_time: datetime`

        Adjusted end of date range.

      - `start_time: datetime`

        Adjusted start of date range.

    - `last_updated: datetime`

      Timestamp of the last dataset update.

    - `normalization: Literal["PERCENTAGE", "MIN0_MAX", "MIN_MAX", 5 more]`

      Normalization method applied to the results. Refer to [Normalization methods](https://developers.cloudflare.com/radar/concepts/normalization/).

      - `"PERCENTAGE"`

      - `"MIN0_MAX"`

      - `"MIN_MAX"`

      - `"RAW_VALUES"`

      - `"PERCENTAGE_CHANGE"`

      - `"ROLLING_AVERAGE"`

      - `"OVERLAPPED_PERCENTAGE"`

      - `"RATIO"`

    - `units: List[MetaUnit]`

      Measurement units for the results.

      - `name: str`

      - `value: str`

  - `top_0: List[Top0]`

    - `name: str`

    - `value: int`

# User Agents

## Get top user agents on robots.txt files

`radar.robots_txt.top.user_agents.directive(UserAgentDirectiveParams**kwargs)  -> UserAgentDirectiveResponse`

**get** `/radar/robots_txt/top/user_agents/directive`

Retrieves the top user agents on robots.txt files.

### Parameters

- `date: Optional[SequenceNotStr[Union[null, null]]]`

  Filters results by the specified array of dates.

- `directive: Optional[Literal["ALLOW", "DISALLOW"]]`

  Filters results by robots.txt directive.

  - `"ALLOW"`

  - `"DISALLOW"`

- `domain_category: Optional[SequenceNotStr[str]]`

  Filters results by domain category.

- `format: Optional[Literal["JSON", "CSV"]]`

  Format in which results will be returned.

  - `"JSON"`

  - `"CSV"`

- `limit: Optional[int]`

  Limits the number of objects returned in the response.

- `name: Optional[SequenceNotStr[str]]`

  Array of names used to label the series in the response.

- `user_agent_category: Optional[Literal["AI"]]`

  Filters results by user agent category.

  - `"AI"`

### Returns

- `class UserAgentDirectiveResponse: …`

  - `meta: Meta`

    Metadata for the results.

    - `confidence_info: Optional[MetaConfidenceInfo]`

      - `annotations: List[MetaConfidenceInfoAnnotation]`

        - `data_source: Literal["ALL", "AI_BOTS", "AI_GATEWAY", 22 more]`

          Data source for annotations.

          - `"ALL"`

          - `"AI_BOTS"`

          - `"AI_GATEWAY"`

          - `"BGP"`

          - `"BOTS"`

          - `"CONNECTION_ANOMALY"`

          - `"CT"`

          - `"DNS"`

          - `"DNS_MAGNITUDE"`

          - `"DNS_AS112"`

          - `"DOS"`

          - `"EMAIL_ROUTING"`

          - `"EMAIL_SECURITY"`

          - `"FW"`

          - `"FW_PG"`

          - `"HTTP"`

          - `"HTTP_CONTROL"`

          - `"HTTP_CRAWLER_REFERER"`

          - `"HTTP_ORIGINS"`

          - `"IQI"`

          - `"LEAKED_CREDENTIALS"`

          - `"NET"`

          - `"ROBOTS_TXT"`

          - `"SPEED"`

          - `"WORKERS_AI"`

        - `description: str`

        - `end_date: datetime`

        - `event_type: Literal["EVENT", "GENERAL", "OUTAGE", 3 more]`

          Event type for annotations.

          - `"EVENT"`

          - `"GENERAL"`

          - `"OUTAGE"`

          - `"PARTIAL_PROJECTION"`

          - `"PIPELINE"`

          - `"TRAFFIC_ANOMALY"`

        - `is_instantaneous: bool`

          Whether event is a single point in time or a time range.

        - `linked_url: str`

        - `start_date: datetime`

      - `level: int`

        Provides an indication of how much confidence Cloudflare has in the data.

    - `date_range: List[MetaDateRange]`

      - `end_time: datetime`

        Adjusted end of date range.

      - `start_time: datetime`

        Adjusted start of date range.

    - `last_updated: datetime`

      Timestamp of the last dataset update.

    - `normalization: Literal["PERCENTAGE", "MIN0_MAX", "MIN_MAX", 5 more]`

      Normalization method applied to the results. Refer to [Normalization methods](https://developers.cloudflare.com/radar/concepts/normalization/).

      - `"PERCENTAGE"`

      - `"MIN0_MAX"`

      - `"MIN_MAX"`

      - `"RAW_VALUES"`

      - `"PERCENTAGE_CHANGE"`

      - `"ROLLING_AVERAGE"`

      - `"OVERLAPPED_PERCENTAGE"`

      - `"RATIO"`

    - `units: List[MetaUnit]`

      Measurement units for the results.

      - `name: str`

      - `value: str`

  - `top_0: List[Top0]`

    - `name: str`

    - `value: int`

    - `fully: Optional[int]`

    - `partially: Optional[int]`

### Example

```python
import os
from cloudflare import Cloudflare

client = Cloudflare(
    api_token=os.environ.get("CLOUDFLARE_API_TOKEN"),  # This is the default and can be omitted
)
response = client.radar.robots_txt.top.user_agents.directive()
print(response.meta)
```

#### Response

```json
{
  "result": {
    "meta": {
      "confidenceInfo": {
        "annotations": [
          {
            "dataSource": "ALL",
            "description": "Cable cut in Tonga",
            "endDate": "2019-12-27T18:11:19.117Z",
            "eventType": "EVENT",
            "isInstantaneous": true,
            "linkedUrl": "https://example.com",
            "startDate": "2019-12-27T18:11:19.117Z"
          }
        ],
        "level": 0
      },
      "dateRange": [
        {
          "endTime": "2022-09-17T10:22:57.555Z",
          "startTime": "2022-09-16T10:22:57.555Z"
        }
      ],
      "lastUpdated": "2019-12-27T18:11:19.117Z",
      "normalization": "PERCENTAGE",
      "units": [
        {
          "name": "*",
          "value": "requests"
        }
      ]
    },
    "top_0": [
      {
        "name": "GPTBot",
        "value": 273,
        "fully": 235,
        "partially": 38
      }
    ]
  },
  "success": true
}
```

## Domain Types

### User Agent Directive Response

- `class UserAgentDirectiveResponse: …`

  - `meta: Meta`

    Metadata for the results.

    - `confidence_info: Optional[MetaConfidenceInfo]`

      - `annotations: List[MetaConfidenceInfoAnnotation]`

        - `data_source: Literal["ALL", "AI_BOTS", "AI_GATEWAY", 22 more]`

          Data source for annotations.

          - `"ALL"`

          - `"AI_BOTS"`

          - `"AI_GATEWAY"`

          - `"BGP"`

          - `"BOTS"`

          - `"CONNECTION_ANOMALY"`

          - `"CT"`

          - `"DNS"`

          - `"DNS_MAGNITUDE"`

          - `"DNS_AS112"`

          - `"DOS"`

          - `"EMAIL_ROUTING"`

          - `"EMAIL_SECURITY"`

          - `"FW"`

          - `"FW_PG"`

          - `"HTTP"`

          - `"HTTP_CONTROL"`

          - `"HTTP_CRAWLER_REFERER"`

          - `"HTTP_ORIGINS"`

          - `"IQI"`

          - `"LEAKED_CREDENTIALS"`

          - `"NET"`

          - `"ROBOTS_TXT"`

          - `"SPEED"`

          - `"WORKERS_AI"`

        - `description: str`

        - `end_date: datetime`

        - `event_type: Literal["EVENT", "GENERAL", "OUTAGE", 3 more]`

          Event type for annotations.

          - `"EVENT"`

          - `"GENERAL"`

          - `"OUTAGE"`

          - `"PARTIAL_PROJECTION"`

          - `"PIPELINE"`

          - `"TRAFFIC_ANOMALY"`

        - `is_instantaneous: bool`

          Whether event is a single point in time or a time range.

        - `linked_url: str`

        - `start_date: datetime`

      - `level: int`

        Provides an indication of how much confidence Cloudflare has in the data.

    - `date_range: List[MetaDateRange]`

      - `end_time: datetime`

        Adjusted end of date range.

      - `start_time: datetime`

        Adjusted start of date range.

    - `last_updated: datetime`

      Timestamp of the last dataset update.

    - `normalization: Literal["PERCENTAGE", "MIN0_MAX", "MIN_MAX", 5 more]`

      Normalization method applied to the results. Refer to [Normalization methods](https://developers.cloudflare.com/radar/concepts/normalization/).

      - `"PERCENTAGE"`

      - `"MIN0_MAX"`

      - `"MIN_MAX"`

      - `"RAW_VALUES"`

      - `"PERCENTAGE_CHANGE"`

      - `"ROLLING_AVERAGE"`

      - `"OVERLAPPED_PERCENTAGE"`

      - `"RATIO"`

    - `units: List[MetaUnit]`

      Measurement units for the results.

      - `name: str`

      - `value: str`

  - `top_0: List[Top0]`

    - `name: str`

    - `value: int`

    - `fully: Optional[int]`

    - `partially: Optional[int]`