Skip to content

AI audio pipelines

Last updated View as MarkdownAgent setup

Use the ai-tts-stt example to connect speech-to-text and text-to-speech services to WebRTC audio. A publisher can generate speech for browser listeners and send microphone audio for transcription.

The application uses Workers AI, a Worker, Durable Objects, and Realtime SFU's WebSocket adapters. The browser uses WebRTC for audio and an application WebSocket for transcription results.

Open the example

The example is experimental. It keeps SFU and AI credentials on the server, but its publisher and cleanup operations are unauthenticated. Add application authentication, authorization, and resource controls before sharing a deployment publicly.

Follow the audio

The application uses separate adapters for the two audio directions:

Publisher text
TTS Durable Object
PCM over WebSocket
CloudflareRealtime SFU
WebRTC audio
Browser listeners
Workers AI speech generation
Browser microphone
WebRTC audio
CloudflareRealtime SFU
PCM over WebSocket
STT Durable Object
Transcript WebSocket
Browser transcript
Workers AI transcription
Publisher text→TTS Durable Object
TTS Durable Object↔Workers AI speech generation
TTS Durable ObjectPCM over WebSocket →Realtime SFU
Realtime SFUWebRTC audio →Browser listeners
Browser microphoneWebRTC audio →Realtime SFU
Realtime SFUPCM over WebSocket →STT Durable Object
STT Durable Object↔Workers AI transcription
STT Durable ObjectTranscript WebSocket →Browser transcript

TTSAdapter and STTAdapter are application Durable Objects. They call Workers AI, manage application state, and convert audio to or from the SFU adapter's 48 kHz stereo PCM format.

The SFU uses separate WebSocket ingest and egress adapters for the two audio directions. Transcripts use a separate application WebSocket. Refer to WebSocket media formats.

Requirements

Use a Cloudflare account with Workers, Durable Objects, Workers AI access, and a Realtime SFU app. The example also needs Node.js and a Workers AI API token for that account. Use the Workers AI token template, or grant Workers AI - Read and Workers AI - Edit permissions.

Deploy the Worker for end-to-end audio. The SFU needs a publicly reachable WebSocket endpoint; localhost development is useful for browser UI changes.

Run the example

  1. Prepare the checkout. Run:

    git clone https://github.com/cloudflare/realtime-examples.git
cd realtime-examples/ai-tts-stt
    npm install

  2. Configure the application. In the checked-in Wrangler configuration, set CF_ACCOUNT and REALTIME_SFU_APP_ID to your account and SFU app identifiers. Follow the configuration guide ↗ for the model and audio-processing configuration.

    Add the backend secrets through Wrangler's interactive prompts:

    npx wrangler secret put CF_API_TOKEN
    npx wrangler secret put REALTIME_SFU_BEARER_TOKEN

  3. Build and deploy. Build the browser assets, then deploy the Worker:

    npm run build:web
    npx wrangler deploy

  4. Connect a publisher and listener. Choose an application session name, such as audio-demo. This is a name you choose, separate from an SFU-generated session ID.

    Open https://<WORKER_HOSTNAME>/audio-demo/publisher and select Publish Session. In another tab, open https://<WORKER_HOSTNAME>/audio-demo/player. Select Connect and wait for the connection. Use the same application name in both URLs.

  5. Generate speech. In the publisher tab, enter text and select Generate Speech. Confirm that the listener plays the generated audio.

  6. Transcribe speech. In the publisher tab, select the STT tab, then Start Mic, and allow microphone access. Wait for the WebRTC connection to become connected, then select Start Forwarding. Speak and observe the returned transcript.

The example's deployment and use guide ↗ describes the controls and URL patterns in detail.

Stop and clean up

To stop the example's flows, select Stop Forwarding, Stop Mic, and Unpublish as applicable. Review the example's Stop the Session and Debug Cleanup instructions ↗ before deleting the backend or exposing administrative operations. Resource ownership and failure recovery need validation for your integration.

Adapt the example

For a spoken tutor, connect finalized transcripts to your dialogue logic and send its responses to TTS. Add turn-taking, interruption handling, conversation state, and authorization for those actions. Keep SFU and AI credentials on the Worker as you adapt the application.

Inspect the implementation

Follow the TTS implementation guide ↗ for speech generation, conversion, and publication. The STT implementation guide ↗ covers microphone forwarding and transcript delivery.

Troubleshooting

Use adapter troubleshooting to diagnose endpoint and media-format failures.

Try another example

For video processing, the WebRTC-to-JPEG example ↗ demonstrates an SFU video track delivered as JPEG frames to a Worker.

Was this helpful?