Crawl

Crawl websites.

browser_rendering.crawl.create() -> CrawlCreateResponse

POST/accounts/{account_id}/browser-rendering/crawl

Get crawl result.

browser_rendering.crawl.get(, ) -> CrawlGetResponse

GET/accounts/{account_id}/browser-rendering/crawl/{job_id}

Cancel a crawl job.

browser_rendering.crawl.delete(, ) -> CrawlDeleteResponse

DELETE/accounts/{account_id}/browser-rendering/crawl/{job_id}

ModelsExpand Collapse

str

Crawl job ID.

class CrawlGetResponse: …

id: str

Crawl job ID.

browser_seconds_used: float

Total seconds spent in browser so far.

finished: float

Total number of URLs that have been crawled so far.

records: List[Record]

List of crawl job records.

metadata: RecordMetadata

status: float

HTTP status code of the crawled page.

url: str

Final URL of the crawled page.

title: Optional[str]

Title of the crawled page.

status: Literal["queued", "errored", "completed", 3 more]

Current status of the crawled URL.

One of the following:

"queued"

"errored"

"completed"

"disallowed"

"skipped"

"cancelled"

url: str

Crawled URL.

html: Optional[str]

HTML content of the crawled URL.

json: Optional[Dict[str, Optional[object]]]

JSON of the content of the crawled URL.

markdown: Optional[str]

Markdown of the content of the crawled URL.

skipped: float

Total number of URLs that were skipped due to include/exclude/subdomain filters. Skipped URLs are included in records but are not counted toward total/finished.

status: str

Current crawl job status.

total: float

Total current number of URLs in the crawl job.

cursor: Optional[str]

Cursor for pagination.

class CrawlDeleteResponse: …

job_id: str

The ID of the cancelled job.

message: str

Cancellation confirmation message.