Skip to content
Cloudflare Docs

Getting started

This guide will instruct you through:

  • Creating your first R2 bucket and enabling its data catalog.
  • Creating an API token needed for pipelines to authenticate with your data catalog.
  • Creating your first pipeline with a simple ecommerce schema that writes to an Apache Iceberg ↗ table managed by R2 Data Catalog.
  • Sending sample ecommerce data via HTTP endpoint.
  • Validating data in your bucket and querying it with R2 SQL.

Prerequisites

  1. Sign up for a Cloudflare account ↗.
  2. Install Node.js ↗.

Node.js version manager

Use a Node version manager like Volta ↗ or nvm ↗ to avoid permission issues and change Node.js versions. Wrangler, discussed later in this guide, requires a Node version of 16.17.0 or later.

1. Create an R2 bucket

  1. If not already logged in, run:

    npx wrangler login

  2. Create an R2 bucket:

    npx wrangler r2 bucket create pipelines-tutorial

2. Enable R2 Data Catalog

Enable the catalog on your R2 bucket:

npx wrangler r2 bucket catalog enable pipelines-tutorial

When you run this command, take note of the "Warehouse" and "Catalog URI". You will need these later.

3. Create an API token

Pipelines must authenticate to R2 Data Catalog with an R2 API token that has catalog and R2 permissions.

  1. In the Cloudflare dashboard, go to the R2 object storage page.

    Go to Overview

  2. Select Manage API tokens.

  3. Select Create Account API token.

  4. Give your API token a name.

  5. Under Permissions, choose the Admin Read & Write permission.

  6. Select Create Account API Token.

  7. Note the Token value.

4. Create a pipeline

First, create a schema file that defines your ecommerce data structure:

Create schema.json:

{
  "fields": [
    {
      "name": "user_id",
      "type": "string",
      "required": true
    },
    {
      "name": "event_type",
      "type": "string",
      "required": true
    },
    {
      "name": "product_id",
      "type": "string",
      "required": false
    },
    {
      "name": "amount",
      "type": "float64",
      "required": false
    }
  ]
}

Use the interactive setup to create a pipeline that writes to R2 Data Catalog:

Terminal window
npx wrangler pipelines setup

Follow the prompts:

  1. Pipeline name: Enter ecommerce

  2. Stream configuration:

    • Enable HTTP endpoint: yes
    • Require authentication: no (for simplicity)
    • Configure custom CORS origins: no
    • Schema definition: Load from file
    • Schema file path: schema.json (or your file path)

  3. Sink configuration:

    • Destination type: Data Catalog Table
    • R2 bucket name: pipelines-tutorial
    • Namespace: default
    • Table name: ecommerce
    • Catalog API token: Enter your token from step 3
    • Compression: zstd
    • Roll file when size reaches (MB): 100
    • Roll file when time reaches (seconds): 10 (for faster data visibility in this tutorial)

  4. SQL transformation: Choose Use simple ingestion query to use:

    INSERT INTO ecommerce_sink SELECT * FROM ecommerce_stream

After setup completes, note the HTTP endpoint URL displayed in the final output.

5. Send sample data

Send ecommerce events to your pipeline's HTTP endpoint:

Terminal window
curl -X POST https://{stream-id}.ingest.cloudflare.com \
  -H "Content-Type: application/json" \
  -d '[
    {
      "user_id": "user_12345",
      "event_type": "purchase",
      "product_id": "widget-001",
      "amount": 29.99
    },
    {
      "user_id": "user_67890",
      "event_type": "view_product",
      "product_id": "widget-002"
    },
    {
      "user_id": "user_12345",
      "event_type": "add_to_cart",
      "product_id": "widget-003",
      "amount": 15.50
    }
  ]'

Replace {stream-id} with your actual stream endpoint from the pipeline setup.

6. Validate data in your bucket

  1. In the Cloudflare dashboard, go to the R2 object storage page.

  2. Select your bucket: pipelines-tutorial.

  3. You should see Iceberg metadata files and data files created by your pipeline. Note: If you aren't seeing any files in your bucket, try waiting a couple of minutes and trying again.

  4. The data is organized in the Apache Iceberg format with metadata tracking table versions.

7. Query your data using R2 SQL

Set up your environment to use R2 SQL:

Terminal window
export WRANGLER_R2_SQL_AUTH_TOKEN=YOUR_API_TOKEN

Or create a .env file with:

WRANGLER_R2_SQL_AUTH_TOKEN=YOUR_API_TOKEN

Where YOUR_API_TOKEN is the token you created in step 3. For more information on setting environment variables, refer to Wrangler system environment variables.

Query your data:

Terminal window
npx wrangler r2 sql query "YOUR_WAREHOUSE_NAME" "
SELECT
    user_id,
    event_type,
    product_id,
    amount
FROM default.ecommerce
WHERE event_type = 'purchase'
LIMIT 10"

Replace YOUR_WAREHOUSE_NAME with the warehouse name from step 2.

You can also query this table with any engine that supports Apache Iceberg. To learn more about connecting other engines to R2 Data Catalog, refer to Connect to Iceberg engines.

Learn more

Managing R2 Data Catalogs Enable or disable R2 Data Catalog on your bucket, retrieve configuration details, and authenticate your Iceberg engine.
Try another example Detailed tutorial for setting up a simple fraud detection data pipeline, and generate events for it in Python.
Pipelines Understand SQL transformations and pipeline configuration.