Log and store upload events in R2 with event notifications

Last reviewed: 9 months ago

This example provides a step-by-step guide on using event notifications to capture and store R2 upload logs in a separate bucket.

Prerequisites

To continue, you will need:

A subscription to Workers Paid, required for using queues.

1. Install Wrangler

To begin, refer to Install/Update Wrangler to install Wrangler, the Cloudflare Developer Platform CLI.

2. Create R2 buckets

You will need to create two R2 buckets:

example-upload-bucket: When new objects are uploaded to this bucket, your consumer Worker will write logs.
example-log-sink-bucket: Upload logs from example-upload-bucket will be written to this bucket.

To create the buckets, run the following Wrangler commands:

npx wrangler r2 bucket create example-upload-bucket
npx wrangler r2 bucket create example-log-sink-bucket

3. Create a queue

Event notifications capture changes to data in example-upload-bucket. You will need to create a new queue to receive notifications:

npx wrangler queues create example-event-notification-queue

4. Create a Worker

Before you enable event notifications for example-upload-bucket, you need to create a consumer Worker to receive the notifications.

Create a new Worker with C3 (create-cloudflare CLI). C3 is a command-line tool designed to help you set up and deploy new applications, including Workers, to Cloudflare.

npm create cloudflare@latest -- consumer-worker

yarn create cloudflare@latest consumer-worker

pnpm create cloudflare@latest consumer-worker

For setup, select the following options:

For What would you like to start with?, choose Hello World example.
For Which template would you like to use?, choose Hello World Worker.
For Which language do you want to use?, choose TypeScript.
For Do you want to use git for version control?, choose Yes.
For Do you want to deploy your application?, choose No (we will be making some changes before deploying).

Then, move into your newly created directory:

cd consumer-worker

5. Configure your Worker

In your Worker project's wrangler.toml file, add a queue consumer and R2 bucket binding. The queues consumer bindings will register your Worker as a consumer of your future event notifications and the R2 bucket bindings will allow your Worker to access your R2 bucket.

wrangler.toml
wrangler.json

name = "event-notification-writer"
main = "src/index.ts"
compatibility_date = "2024-03-29"
compatibility_flags = ["nodejs_compat"]

[[queues.consumers]]
queue = "example-event-notification-queue"
max_batch_size = 100
max_batch_timeout = 5

[[r2_buckets]]
binding = "LOG_SINK"
bucket_name = "example-log-sink-bucket"

{
  "name": "event-notification-writer",
  "main": "src/index.ts",
  "compatibility_date": "2024-03-29",
  "compatibility_flags": [
    "nodejs_compat"
  ],
  "queues": {
    "consumers": [
      {
        "queue": "example-event-notification-queue",
        "max_batch_size": 100,
        "max_batch_timeout": 5
      }
    ]
  },
  "r2_buckets": [
    {
      "binding": "LOG_SINK",
      "bucket_name": "example-log-sink-bucket"
    }
  ]
}

6. Write event notification messages to R2

Add a queue handler to src/index.ts to handle writing batches of notifications to our log sink bucket (you do not need a fetch handler):

export interface Env {
  LOG_SINK: R2Bucket;
}

export default {
  async queue(batch, env): Promise<void> {
    const batchId = new Date().toISOString().replace(/[:.]/g, "-");
    const fileName = `upload-logs-${batchId}.json`;

    // Serialize the entire batch of messages to JSON
    const fileContent = new TextEncoder().encode(
      JSON.stringify(batch.messages),
    );

    // Write the batch of messages to R2
    await env.LOG_SINK.put(fileName, fileContent, {
      httpMetadata: {
        contentType: "application/json",
      },
    });
  },
} satisfies ExportedHandler<Env>;

7. Deploy your Worker

To deploy your consumer Worker, run the wrangler deploy command:

npx wrangler deploy

8. Enable event notifications

Now that you have your consumer Worker ready to handle incoming event notification messages, you need to enable event notifications with the wrangler r2 bucket notification create command for example-upload-bucket:

npx wrangler r2 bucket notification create example-upload-bucket --event-type object-create --queue example-event-notification-queue

9. Test

Now you can test the full end-to-end flow by uploading an object to example-upload-bucket in the Cloudflare dashboard. After you have uploaded an object, logs will appear in example-log-sink-bucket in a few seconds.

Was this helpful?

Cloudflare Dashboard Discord Community Learning Center Support Portal

Cookie Settings