Crawl websites.
Starts a crawl job for the provided URL and its children. Check available options like gotoOptions and waitFor* to control page load behaviour.
Security
API Token
The preferred authorization scheme for interacting with the Cloudflare API. Create a token.
API Email + API Key
The previous authorization scheme for interacting with the Cloudflare API, used in conjunction with a Global API key.
The previous authorization scheme for interacting with the Cloudflare API. When possible, use API tokens instead of Global API keys.
Accepted Permissions (at least one required)
Parameters
The maximum duration allowed for the browser action to complete after the page has loaded (such as taking screenshots, extracting content, or generating PDFs). If this time limit is exceeded, the action stops and returns a timeout error.
Maximum number of levels deep the crawler will traverse from the starting URL.
Maximum age of a resource that can be returned from cache in seconds. Default is 1 day.
Unix timestamp (seconds since epoch) indicating to only crawl pages that were modified since this time. For sitemap URLs with a lastmod field, this is compared directly. For other URLs, the crawler will use If-Modified-Since header when fetching. URLs without modification information (no lastmod in sitemap and no Last-Modified header support) will be crawled. Note: This works in conjunction with maxAge - both filters must pass for a cached resource to be used. Must be within the last year and not in the future.
Crawl websites.
import os
from cloudflare import Cloudflare
client = Cloudflare(
api_token=os.environ.get("CLOUDFLARE_API_TOKEN"), # This is the default and can be omitted
)
crawl = client.browser_rendering.crawl.create(
account_id="account_id",
url="https://example.com",
)
print(crawl){
"result": "result",
"success": true,
"errors": [
{
"code": 0,
"message": "message"
}
]
}{
"errors": [
{
"code": 2001,
"message": "Rate limit exceeded"
}
],
"success": false
}Returns Examples
{
"result": "result",
"success": true,
"errors": [
{
"code": 0,
"message": "message"
}
]
}{
"errors": [
{
"code": 2001,
"message": "Rate limit exceeded"
}
],
"success": false
}