This tutorial explains how to analyze Cloudflare Logs using Elastic and Kibana.
If you haven’t used Cloudflare Logs before, visit our logs documentation for more details. Contact your Cloudflare Customer Account Team to enable logs for your account.
Before sending your Cloudflare log data to Elastic, make sure that you:
Cloudflare logs are HTTP/HTTPS request logs in JSON format and are gathered from our 194+ data centers globally. By default, timestamps are returned as Unix nanosecond integers. We recommend using the RFC 3339 format for sending logs to Elastic.
Before getting Cloudflare logs into Elastic:
Install an AWS Lambda function, available in the file cloudflare-elastic-lambda.zip from Github at this location: https://github.com/cloudflare/cloudflare-elastic/releases/tag/v0.1.
Upload the cloudflare-elastic-lambda.zip file to an S3 bucket.
Create an account on Elastic Cloud and log in.
Once logged in, create a new deployment where the Cloudflare logs will reside.
Configure your new deployment with the following parameters:
Click Customize Deployment. On this page, you can set your Elasticsearch cluster memory and storage.
I/O Optimized Template Configuration
For this options, we recommend configuring your cluster to have 2 availability zones and 8 GB of RAM.
Hot-Warm Template Configuration
For this option, we recommend configuring your cluster as:
Hot Zone: 2 availability zones, 8 GB RAM
Warm Zone: 2 availability zones, 8 GB RAM
You are now ready to create your Elastic deployment. Click Create deployment. The page will refresh with details of your new cluster. It is important to save the randomly generated password (see screenshot). All Elastic deployments are secure by default and are bootstrapped with a randomly generated password for the Elastic user. You will use this password to log in to view your Cloudflare logs.
Next, to configure your deployment:
From https://github.com/cloudflare/cloudflare-elastic/releases/tag/v0.1 download the following files:
Using the Elasticsearch endpoint for your new deployment (found on the main page), run the following command from a terminal, using the random password from above.
./install-artifacts.sh -u elastic -p <RANDOM PASSWORD> -e https://_<YOUR DEPLOYMENT ENDPOINT>_:9243
To install the Cloudflare dashboards, log in to the Kibana user interface using the elastic username and the randomly generated password.
Navigate to Management > Kibana > Saved Objects.
Upload dashboards.json by clicking the Import link.
To create the Lambda function:
Install the function: create the Lambda, which will read Cloudflare logs from S3 and import them into your Elastic cluster.
Log in to your AWS console and navigate to the Lambda section. Create a new function, using the Java 8 runtime and give it a name such as cloudflare-elastic-logs.
Configure the function. The Lambda function requires the following environment variables:
elastic_hostname: Copy the Elasticsearch endpoint URL from your deployment. Remove the port number and https:// prefix; for example:
elastic_username: Enter elastic.
To connect your Cloudflare S3 log bucket. the last step is to tell the Lambda function to listen for events on the S3 bucket where your Cloudflare logs reside. Choose the S3 trigger type and configure it with the name of the S3 bucket. For Event type, select All object create events.
Give the policy a name and save it.
Save the Lambda and start logging.
Finally, save the Lambda configuration. Once it is saved, the Cloudflare logs will start showing up in Kibana on your Elastic Cloud deployment.
Once Successfully imported, you can find all Cloudflare dashboards under Kibana > Dashboard.
There are nine dashboards to help you analyze Cloudflare logs. You can also use filters within the dashboards to help narrow the analysis by date and time, device type, country, user agent, client IP, hostname, and more. Filtering can further help you with debugging and tracing.
This is a quick overview of the most important metrics from your Cloudflare Logs, including the total number of requests, top visitors by geography, IP, user agent, traffic type, the total number of threats, and bandwidth usage.
Cloudflare - Security (WAF) - Get insights on threat identification and mitigation by our Web Application Firewall, including events like SQL injections, XSS, and more. Use this data to fine tune the firewall to target obvious threats and prevent false positives.
Cloudflare - Security (Rate Limiting) - Get insights on rate limiting protection against denial-of-service attacks, brute-force login attempts, and other types of abusive behavior targeted at your websites or applications.
Cloudflare - Security (Bot Management) - Reliably detects and mitigates bad bots to prevent credential stuffing, spam registration, content scraping, click fraud, inventory hoarding, and other malicious activities.
Cloudflare - Performance (Requests, Bandwidth, Cache) - Identify and address performance issues and caching misconfigurations. Metrics include total vs. cached bandwidth, saved bandwidth, total requests, cache ratio, top uncached requests, and more.
Cloudflare - Performance (Hostname, Content Type, Request Methods, Connection Type) - Get insights into your most popular hostnames, most requested content types, breakdown of request methods, and connection type.
Get insights on the availability of your websites and applications. Metrics include origin response error ratio, origin response status over time, percentage of 3xx/4xx/5xx errors over time, and more.
All dashboard have a set of filters that you can apply to the entire dashboard, as shown in the following example. Filters are applied across the entire dashboard.
You can use filters to drill down and examine the data at a granular level. Filters include client country, client device type, client IP, client request host, client request URI, client request user agent, edge response status, origin IP, and origin response status.
The default time interval is set to 24 hours. Note that for correct calculations filter will need to exclude Worker subrequests (WorkerSubrequest = false) and purge requests (ClientRequestMethod is not PURGE).
Edge Response Status
Origin Response Status
Client Request Method
If you detect issues with your AWS Lambda function in Elastic, you can review the AWS CloudWatch logs that are generated automatically for tht specific Lambda function.
To begin, in AWS:
A list of log streams generated by the Lambda function (see image below) appears. Each stream contains log messages. However, some log streams will either be empty or not contain any useful information. You might need to review several of them.
When you click to review a stream, you want to look for messages starting with the the text Connected to cluster: and ending with status: [GREEN] as shown in the image below.
If you see status: [RED], then your cluster isn’t healthy and it’s likely that your Cloudflare logs won’t appear. If this is the case, review how to debug in Kibana (see below).
It is important to verify the return status of the call to Elasticsearch. Lines starting with Flushing [N] logs to elasticsearch and followed by a response line indicate that everything is working as expected.
You might see a WARNING message containing text that says ‘y’ year should be replaced…. You can ignore this message.
If you run into any other issues, take note of the exact return message and contact your Cloudflare support team.
To analyze the health status of the Lambda function from Kibana:
The first column should read green. If it does not, or if there are no cloudflare-<DATE> indices, then there is a problem loading the logs from the AWS Lambda function.