Understand site traffic
3 min read
Another characteristic to consider is your application’s traffic. Several aspects of your traffic might affect how you implement Bot management.
If you are experiencing bot attacks, consider the nature of the attack.
These types of actions fall into two general categories:
A bot submitting data through a form
- Credential stuffing on login endpoints
- Content spam
- Inventory hoarding
- Credit card stuffing
A bot viewing data intended for human eyes only
- Content scraping
- Ad click fraud
- Email address harvesting
If bots are submitting data through forms, you should likely be creating focused rules to block traffic on specific endpoints.
If bots are viewing data intended for human eyes only, you may want broader rules limiting bot interactions across your application.
Cloudflare maintains an internal list of Verified Bots that are associated with search engine optimization (SEO), website monitoring, and more.
You can use this list to prevent any bot protection measures from impacting otherwise helpful bots, such as search crawlers.
To allow verified bots, you would need to include
not (cf.bot_management.verified_bot) as part of a custom rule.
By default, Bot management rules will block non-browser traffic.
This can be problematic if your application receives a lot of API traffic, or if you have tools related to:
- Indexing content for search.
- Auditing content (links, headers, etc.).
- Monitoring uptime.
- Forward proxying traffic, such as secure web gateways.
Depending on your application, you may want to write rules that allow specific types of automated traffic or rules that allow all automated traffic to specific endpoints (
/api, for example).
Pay specific attention to:
- Which endpoints are being targeted.
- The top non-Mozilla user agents.
- Traffic from Outlook or Office user-agents.
- Traffic from cloud-based Secure Web Gateways (ASNs labeled with the proxy provider).
- Traffic from on-premises forward proxies.
- Whether requests come from a predictable IP address and ASN, or have a similar .
Mobile app traffic
Because of how mobile applications send requests, Bot Management has the potential to score mobile traffic differently than browser-based traffic.
You can generally identify mobile traffic with common user agent strings, though these strings may differ between iOS and Android. Malicious actors might also try to impersonate your mobile application traffic with user agent strings.