Monitor and batch your website data

Watch this episode

Workflows can be used to process batches of data, ensuring each item in the batch goes through a defined process with reliable execution. This section demonstrates processing a batch of puns using the Punderful application as an example.

Chapters

• 
  Introduction 3s
• 
  Implementing Workflows with Puns Dataset 1m29s
• 
  Deployment and Monitoring 2m52s
• 
  Admin Dashboard and Further Insights 4m0s

Related content

If you want to dive into detail, refer to the following pages:

• Source code for the Punderful repository ↗
• Cloudflare Workflows
• Cloudflare Workers AI

Step-by-step tutorial

The Punderful application processes user-submitted puns by performing content moderation, creating embeddings, categorizing, and adding them to a vector store. This process is defined as a Workflow. To process a batch of existing puns (from an open-source dataset called OPun), a batch endpoint is created that iterates through the puns and triggers the defined Workflow for each one.

Batch Processing Code

The following code snippet shows the endpoint responsible for batch processing:

See here ↗

This code:

1. Fetches the list of puns from a JSON file (puns.json).
2. Logs the number of puns being processed.
3. Sets a user ID for tracking.
4. Loops through each pun.
5. Performs basic text cleaning on the pun.
6. Inserts the pun into the database (handled by insertPun).
7. Triggers the PUBLISH Workflow for each individual pun using c.env.PUBLISH.create(). The Workflow is given a unique ID using crypto.randomUUID().

Monitoring Workflow Instances via CLI

The Cloudflare Wrangler CLI provides commands to monitor and manage Workflows and their instances.

To list the available workflows associated with your account:

npx wrangler workflows list

To list the instances of a specific workflow (for example, the publish workflow):

npx wrangler workflows instances list publish

This command will show a list of workflow instances, their status (Queued, Running, Completed, Errored), and timestamps.

To view the details of a specific workflow instance, including its steps and their status, duration, and output:

npx wrangler workflows instances describe publish <instance-id>

Replace <instance-id> with the actual ID of a running or completed instance from the list command output.

Example CLI Output (Describe Instance)

Describing a workflow instance provides a detailed breakdown of its execution:

Workflow Name: publish
Instance ID: oPun-batch-aea07d75-95fa-448f-9573-6e435388eff7
Version ID: 75665fce-24a1-4c83-a561-088aabc91e5f
Status: Completed
Trigger: API
Queued: 10/24/2024, 1:43:45 AM
Success: Yes
Start: 10/24/2024, 1:43:45 AM
End: 10/24/2024, 1:43:49 AM
Duration: 4 seconds
Last Successful Step: update-status-to-published-1
Steps:

Name: content-moderation-1
Type: Step
Start: 10/24/2024, 1:43:45 AM
End: 10/24/2024, 1:43:45 AM
Duration: 0 seconds
Success: Yes
Output: "true"
Config: {"retries":{"limit":5,"delay":1000,"backoff":"exponential"},"timeout":"10 minutes"}
Attempts:
  Status: Completed
  Start Time: Oct 23, 2024 6:44:57 PM
  End Time: Oct 23, 2024 6:44:57 PM
  Wall Time: 180 ms
... (additional steps like create-pun-embedding-1, categorize-pun-1, add-embeddings-to-vector-store-1, update-status-to-published-1)

This output shows the status, start/end times, duration, success status, and even the output and configuration for each step within the workflow instance.

Monitoring Workflow Instances via Cloudflare Dashboard

You can also monitor Workflows and their instances directly in the Cloudflare Dashboard.

This dashboard view provides a user-friendly way to observe the progress of your batch jobs, identify failed instances, and inspect the execution details of each step.

Series overview

Watch Episode 1: Understand Cloudflare Workflows In this episode, we introduce Cloudflare Workflows, which provides durable execution capabilities, allowing developers to create reliable, repeatable workflows that run in the background.

Watch Episode 2: Monitor and batch your website data In this episode, we describe how Workflows can be used to process batches of data, ensuring each item in the batch goes through a defined process with reliable execution.

Watch Episode 3: Use cron triggers to develop time-aware applications In this episode, we review Workflows ability to explicitly schedule tasks using cron triggers and pause execution with `step.sleep` allows developers to build sophisticated, time-aware applications.