Bot Management Dashboard

This tutorial explains how to activate Cloudflare Bot Management and view related dashboards in the Elastic, Google Data Studio, Looker, Splunk, and Sumo Logic analytics platforms.


Overview

Bot Management is a Cloudflare Enterprise feature. To enable it, contact your Cloudflare Account Team.

If you haven’t used Cloudflare Logs before, visit our logs documentation for more details. Contact your Cloudflare Customer Account Team to enable logs for your account.

Prerequisites

Before getting started, make sure that you:

  • Have a Cloudflare Enterprise account with Cloudflare Logs and Bot Management enabled
  • Configure Logpush or Logpull
  • Have enabled the EdgePathingSrc and EdgePathingStatus fields in Cloudflare Logs to ensure bot requests are captured
  • Are familiar with Cloudflare Bot Management
  • Follow your analytics provider’s guides for getting logs from AWS S3 or Google Cloud Platform into your analytics platform

Task 1 - Configure Cloudflare to Detect Bot Traffic

Before proceeding, make sure that you’ve enabled Cloudflare Bot Management for your Enterprise account. To start:
  1. In the Cloudflare dashboard, create a Firewall rule. Learn more.
  2. For Rule name, enter Bot management - Generic.
  3. Under Expression Preview, click Edit Expression, then copy and paste the following expression: (bot score lt 30 and not verified_bot).
    • This rule only selects requests with a bot score less than 30 and excludes good bots. Requests with as score under 30 are considered bad bots. Your Firewall rule can be as granular as required. For example, applying conditions only for a specific URL, like login or sign-up pages.
  4. Choose an action: Log or Challenge (Captcha).
    • Cloudflare recommends starting with the action of Log and running it for several days in order to identify which requests fall under the rule above to check if any false positives are registered. You can refine the rule expression further, based on your findings. Once you complete testing, switch the rule action to Challenge (Captcha).
  5. Click Save.

You can track bots in Cloudflare logs only when using the action Challenge (Captcha). Cloudflare plans to add support for the Log action in late 2019.

Create Cloudflare Firewall rule to identify bad bots


Task 2 - View Bot Traffic Dashboards

Viewing dashboards for existing analytics integrations

The Cloudflare Bot Management Dashboard is already available in Elastic, Google Data Studio, Looker, Splunk, and Sumo Logic. If you use any of those platforms, you do not need to do anything else to view these existing dashboards.

See the Analytics Integrations and Analyze log data with Google Cloud sections for details.

Viewing dashboards for other platforms

If you use a platform that is not part of the integrations mentioned above, you will need to design your own dashboards.

The following sub-sections describe how to identify bot requests and perform the correct calculations using data from your Cloudflare logs.

Identify bot requests

Combining the following two fields reveals if the request is a bot request:

EdgePathingSrc = 'filterBasedFirewall' AND EdgePathingStatus = 'captchaNew'
Identify valid requests

Valid requests are those for which the user can solve the CAPTCHA. In this case, EdgePathingStatus = 'captchaSucc'.

The following combination of fields reveals which CAPTCHAs were solved:

EdgePathingSrc = 'filterBasedFirewall' AND EdgePathingStatus = 'captchaSucc'
Calculate number of bad bots

To start, make sure to exclude solved CAPTCHAs from calculations:

Bad Bots = SUM(EdgePathingSrc = 'filterBasedFirewall' AND EdgePathingStatus = 'captchaNew') - SUM(EdgePathingSrc = 'filterBasedFirewall' AND EdgePathingStatus = 'captchaSucc' )

If you see a significant increase in CAPTCHA conversion rate, contact your Cloudflare Account Team for further investigation.

Understand the Bot Management Dashboard

The Bot Management Dashboard consists of three columns to help you better understand and monitor traffic behavior. You can also analyze the ratio between all traffic and bot traffic.

The columns are:

  1. Global Traffic - Shows all requests

  2. Solved CAPTCHAs - Shows requests with solved CAPTCHAs

  3. Bad Bots - Shows confirmed bad bot requests, which exclude solved CAPTCHAs

Limitations: To identify bot requests correctly, the associated Firewall rules can only be set to the action Challenge (Captcha). If you have more than one rule with this same action (and used for purposes other than bot management), such rule will interfere and you will see overlapping or conflicting metrics without the option to identify which Firewall rule triggered.

Example dashboards

The following images demonstrate some of the dashboards available.

Elastic (Learn more) Cloudflare Bot Management in Elastic

Google Data Studio (Learn more) Cloudflare Bot Management in Google Data Studio page 1 Cloudflare Bot Management in Google Data Studio page 2

Graylog (Learn more) Cloudflare Bot Management in Graylog

Looker (Learn more) Cloudflare Bot Management in Looker

Splunk (Learn more) Cloudflare Bot Management in Splunk page 1 Cloudflare Bot Management in Splunk page 2

Sumo Logic (Learn more) Cloudflare Bot Management in Sumo Logic