Total number of URLs that have been crawled so far.
records: array of object {metadata, status, url, 3 more}
List of crawl job records.
metadata: object {status, url, title}
status: number
HTTP status code of the crawled page.
url: string
Final URL of the crawled page.
title: optional string
Title of the crawled page.
status: "queued" or "errored" or "completed" or 3 more
Current status of the crawled URL.
One of the following:
"queued"
"errored"
"completed"
"disallowed"
"skipped"
"cancelled"
url: string
Crawled URL.
html: optional string
HTML content of the crawled URL.
json: optional map[unknown]
JSON of the content of the crawled URL.
markdown: optional string
Markdown of the content of the crawled URL.
skipped: number
Total number of URLs that were skipped due to include/exclude/subdomain filters. Skipped URLs are included in records but are not counted toward total/finished.
