URL Scanner
To better understand Internet usage around the world, use Cloudflare's URL Scanner. With Cloudflare's URL Scanner, you have the ability to investigate the details of a domain, IP, URL, or ASN. Cloudflare's URL Scanner is available in the Security Center of the Cloudflare dashboard, Cloudflare Radar ↗ and the Cloudflare API.
To make your first URL scan using the API, you must obtain a URL Scanner specific API token. Create a Custom Token with Account > URL Scanner in the Permissions group, and select Edit as the access level.
Once you have the token, and you know your account_id, you are ready to make your first request to the API at https://api.cloudflare.com/client/v4/accounts/{account_id}/urlscanner/.
To submit a URL to scan, the only required information is the URL to be scanned in the POST request body:
curl "https://api.cloudflare.com/client/v4/accounts/{account_id}/urlscanner/scan" \--header "Authorization: Bearer <API_TOKEN>" \--header "Content-Type: application/json" \--data '{ "url": "https://www.example.com"}'By default, the report will have a Public visibility level, which means it will appear in the recent scans ↗ list and in search results. It will also include a single screenshot with desktop resolution.
A successful response will have a status code of 200 and be similar to the following:
{ "errors": [], "messages": [{ "message": "Submission successful" }], "result": { "time": "2022-09-15T00:00:00Z", "url": "https://www.example.com", "uuid": "095be615-a8ad-4c33-8e9c-c7612fbf6c9f", "visibility": "Public" }, "success": true}The result.uuid property in the response above identifies the scan and will be required when fetching the scan report.
Here's an example request body with some custom configuration options:
{ "url": "https://example.com", "screenshotsResolutions": [ "desktop", "mobile", "tablet" ], "customHeaders": { "user-agent": "My-custom-user-agent", }, "visibility": "Unlisted"}Above, the visibility level is set as Unlisted, which means that the scan report won't be included in the recent scans ↗ list nor in search results. In effect, only users with knowledge of the scan ID will be able to access it.
There will also be three screenshots taken of the webpage, one per target device type. The User-Agent ↗ HTTP Header will be set as "My-custom-user-agent". Note that you can set any custom HTTP header, including Authorization ↗.
Once the URL Scan submission is made, the current progress can be checked by calling https://api.cloudflare.com/client/v4/accounts/{account_id}/urlscanner/scan/{scan_id}. The scan_id will be the result.uuid value returned in the previous response.
While the scan is in progress, the HTTP status code will be 202, once it's finished it will be 200. Clients are advised to poll every 10-30 seconds.
The response will include, among others, the following top properties in result.scan:
task- Information on the scan submission.page- Information pertaining to the primary request (for example, response cookies) and the webpage itself (e.g. console messages).meta- Meta processors output including detected technologies, categories, rank and others.ips- IPs contacted.asns- AS Numbers contacted.geo- GeoIP information derived from contacted IPs.domains- Hostnames contacted, includingdnsrecord information.links- Outgoing links detected in the DOM.performance- Timings as given by thePerformanceNavigationTiming↗ interface.certificates- TLS certificates of HTTP responses.verdicts- Verdicts on malicious content.
Some examples of more specific properties include:
task.uuid- ID of the scan.task.effectiveUrl- URL of the primary request, after all HTTP redirects.task.success- Whether scan was successful or not. Scans can fail for various reasons, including DNS errors.task.status- Current scan status, for example,Queued,InProgress, orFinished.meta.processors.categories- Cloudflare categories of the main hostname contacted.meta.processors.securityRiskCategories- Cloudflare categories, representing a security risk, of the main hostname contacted.meta.processors.phishing- What kind of phishing, if any, was detected.meta.processors.rank- Cloudflare Radar Rank ↗ of the main hostname contacted.meta.processors.tech- What kind of technologies were detected as being in use by the website, with the help of Wappalyzer ↗.page.country- GeoIP country name of the main IP address contacted.page.cookies- Cookies set by the page.page.console- JavaScript console messagespage.js.variables- Non-standard JavaScript global variables.page.securityViolations- CSP or SRI ↗ violations.verdicts.overall.malicious- Whether the website was considered malicious at the time of the scan. Please check the remaining properties for each subsystem(s) for specific threats detected.
The Get URL Scan API endpoint documentation contains the full response schema.
To fetch the scan's screenshots or full network log refer to the corresponding endpoints' documentation.
Public scans can also be searched for. To search for scans to the hostname google.com, use the query parameter page_hostname=google.com:
curl "https://api.cloudflare.com/client/v4/accounts/{account_id}/urlscanner/scan?page_hostname=google.com" \--header "Authorization: Bearer <API_TOKEN>"Search results will also include your own Unlisted scans.
If, instead, you wanted to search for scans that made at least one request to the hostname cdnjs.cloudflare.com, for example sites that use a JavaScript library hosted at cdnjs.cloudflare.com, use the query parameter hostname=cdnjs.cloudflare.com:
curl "https://api.cloudflare.com/client/v4/accounts/{account_id}/urlscanner/scan?hostname=cdnjs.cloudflare.com" \--header "Authorization: Bearer <API_TOKEN>"You can also search for the hash in the URL Scanner API.
Go to Search URL scans in the API documentation for the full list of available options.
Alternatively, you can search for the hash on the Cloudflare dashboard ↗ by selecting your account > Security Center > Investigate > Enter the hash > Select Search.