Crawl

Crawl websites.

POST/accounts/{account_id}/browser-rendering/crawl

Get crawl result.

GET/accounts/{account_id}/browser-rendering/crawl/{job_id}

Cancel a crawl job.

DELETE/accounts/{account_id}/browser-rendering/crawl/{job_id}

ModelsExpand Collapse

CrawlCreateResponse = string

Crawl job ID.

CrawlGetResponse object { id, browserSecondsUsed, finished, 6 more }

id: string

Crawl job ID.

browserSecondsUsed: number

Total seconds spent in browser so far.

finished: number

Total number of URLs that have been crawled so far.

records: array of object { status, url, html, 4 more }

List of crawl job records.

status: "queued" or "errored" or "completed" or 3 more

Current status of the crawled URL.

One of the following:

"queued"

"errored"

"completed"

"disallowed"

"skipped"

"cancelled"

url: string

Crawled URL.

html: optional string

HTML content of the crawled URL.

json: optional map[unknown]

JSON of the content of the crawled URL.

markdown: optional string

Markdown of the content of the crawled URL.

metadata: optional object { status, url, lastModified, 6 more }

Absent for urls that never reached a fetch.

status: number

HTTP status code of the crawled page.

url: string

Final URL of the crawled page.

lastModified: optional string

Value of the Last-Modified response header, when provided by the origin.

"og:description": optional string

Open Graph description of the crawled page.

"og:image": optional string

Open Graph image of the crawled page.

"og:site_name": optional string

Open Graph site name of the crawled page.

"og:title": optional string

Open Graph title of the crawled page.

"og:type": optional string

Open Graph type of the crawled page.

title: optional string

Title of the crawled page.

outcome: optional "success" or "robots_disallowed" or "crawl_purpose_disallowed" or 9 more

Result of processing the URL. Absent for older records.

One of the following:

"success"

"robots_disallowed"

"crawl_purpose_disallowed"

"content_use_disallowed"

"connection_error"

"timeout"

"browser_unavailable"

"format_generation_error"

"unsupported_content_type"

"unclassified_error"

"page_error"

"too_large"

robots: array of object { origin, state, status }

Robots.txt state for the first 50 origins encountered by the crawl.

origin: string

Origin whose robots.txt was checked.

state: "available" or "absent" or "fetch_failed"

Whether usable robots.txt rules were available.

One of the following:

"available"

"absent"

"fetch_failed"

status: optional number

HTTP status that established the current robots.txt state. Absent when no HTTP response was received.

skipped: number

Total number of URLs that were skipped due to include/exclude/subdomain filters. Skipped URLs are included in records but are not counted toward total/finished.

status: string

Current crawl job status.

total: number

Total current number of URLs in the crawl job.

cursor: optional string

Cursor for pagination.

CrawlDeleteResponse object { job_id, message }

job_id: string

The ID of the cancelled job.

message: string

Cancellation confirmation message.