Bot Management Dashboard
This tutorial explains how to activate Cloudflare Bot Management and view related dashboards in the Elastic, Google Data Studio, Looker, Splunk, and Sumo Logic analytics platforms.
Before getting started, make sure that you:
- Have a Cloudflare Enterprise account with Cloudflare Logs and Bot Management enabled
- Configure or
- Have enabled the fields in Cloudflare Logs to ensure bot requests are captured
- Are familiar with
- Follow your analytics provider’s guides for getting logs from AWS S3 or Google Cloud Platform into your analytics platform
Task 1 - Configure Cloudflare to Detect Bot Traffic
Before proceeding, make sure that you’ve enabled Cloudflare Bot Management for your Enterprise account. To start:
- In the Cloudflare dashboard, create a Firewall rule. .
- For Rule name, enter Bot management - Generic.
- Under Expression Preview, click Edit Expression, then copy and paste the following expression:
(cf.bot_management.score lt 30 and not cf.bot_management.verified_bot).
- This rule only selects requests with a bot score less than 30 and excludes good bots. Requests with a score under 30 are considered bad bots. Your Firewall rule can be as granular as required. For example, applying conditions only for a specific URL, like login or sign-up pages.
- Choose an action: Log or Challenge (Captcha).
- Cloudflare recommends starting with the action of Log and running it for several days in order to identify which requests fall under the rule above to check if any false positives are registered. You can refine the rule expression further, based on your findings. Once you complete testing, switch the rule action to Challenge (Captcha).
- Click Save.
Task 2 - View Bot Traffic Dashboards
Viewing dashboards for existing analytics integrations
The Cloudflare Bot Management Dashboard is already available in Elastic, Google Data Studio, Looker, Splunk, and Sumo Logic. If you use any of those platforms, you do not need to do anything else to view these existing dashboards.
Viewing dashboards for other platforms
If you use a platform that is not part of the integrations mentioned above, you will need to design your own dashboards.
The following sub-sections describe how to identify bot requests and perform the correct calculations using data from your Cloudflare logs.
Identify bot requests
Combining the following two fields reveals if the request is a bot request:
EdgePathingSrc = 'filterBasedFirewall' AND EdgePathingStatus = 'captchaNew'
Identify valid requests
Valid requests are those for which the user can solve the CAPTCHA. In this case,
EdgePathingStatus = 'captchaSucc'.
The following combination of fields reveals which CAPTCHAs were solved:
EdgePathingSrc = 'filterBasedFirewall' AND EdgePathingStatus = 'captchaSucc'
Calculate number of bad bots
To start, make sure to exclude solved CAPTCHAs from calculations:
Bad Bots = SUM(EdgePathingSrc = 'filterBasedFirewall' AND EdgePathingStatus = 'captchaNew') - SUM(EdgePathingSrc = 'filterBasedFirewall' AND EdgePathingStatus = 'captchaSucc' )
If you see a significant increase in CAPTCHA conversion rate, contact your Cloudflare Account Team for further investigation.
Understand the Bot Management Dashboard
The Bot Management Dashboard consists of three columns to help you better understand and monitor traffic behavior. You can also analyze the ratio between all traffic and bot traffic.
The columns are:
Global Traffic - Shows all requests
Solved CAPTCHAs - Shows requests with solved CAPTCHAs
Bad Bots - Shows confirmed bad bot requests, which exclude solved CAPTCHAs
The following images demonstrate some of the dashboards available.