How Pub/Sub works
Cloudflare Pub/Sub is a powerful way to send (publish) messages to and from remote clients.
There are four major concepts to understand with Pub/Sub:
Brokers and namespaces
Brokers and namespaces are fundamentally “containers” for organizing clients, topics, and their associated permissions.
A namespace is a collection of brokers that can be organized by location, end-customer, environment (production vs. staging), or by teams within an organization. When starting out, one namespace is typically all you need. Namespaces are globally unique across all customers.
A broker is a term commonly used in MQTT to refer to the “server,” but because an MQTT “server” is effectively a relay or proxy that accepts messages from one set of clients and sends them to the next, the term broker is used to distinguish from a typical client-server architecture. Clients – and their credentials – are scoped to a broker, and a broker itself is an addressable endpoint that accepts MQTT connections from clients on a TCP port.
For example, you could create a namespace called
acme-telemetry and a broker called
dev-broker. Together, these define an endpoint of
dev-broker.acme-telemetry.cloudflarepubsub.com that authenticated clients can connect, send (publish), and receive (subscribe) messages against.
All clients must authenticate – prove they are allowed to connect – to a broker, and credentials are scoped per broker. A client with credentials for
dev-broker.acme-telemetry.cloudflarepubsub.com would need a separate set of credentials to connect to
prod-broker.acme-telemetry.cloudflarepubsub.com even if both are in the same account.
- Authentication is based on the MQTT standard, which allows for username and password (often called token auth) authentication, as well as implementation specific Mutual TLS (often called TLS Client Credentials) based authentication.
- With Cloudflare Pub/Sub, the easiest way to get started is to issue per-client tokens. Tokens take the place of the password in the authentication flow. These tokens are signed , which can only be generated by Cloudflare. Because they are signed, the client ID, permissions, or other claims embedded in the token cannot be changed without invalidating the signature.
Topics and subscriptions
The topic is the core concept of Pub/Sub and MQTT, and all messages are contained within the topic they are published on. In MQTT, topics are strings that are separated by a
/ (forward slash) character to denote different topic levels and define a hierarchy for subscribers. Importantly, and one of the benefits of the underlying MQTT protocol, topics do not have to be defined centrally. Clients can publish to arbitrary topics, provided the broker allows that client to do so, which allows you to flexibly group messages as needed.
A Pub/Sub client can be both a publisher and subscriber at once, and can publish and subscribe to multiple topics at once.
As a set of best practices when constructing and naming your topics:
- Define topics as a consistent hierarchy such as location/data-type/data-format/. For example, a client in the EU publishing HTTP metrics data would publish to
eu/metrics/request_countso that subscribers can more easily identify the data.
- Ensure that you are consistent with your casing (lower vs. upper case) for topic names. Topics are case-sensitive in the MQTT protocol and a client subscribed to
eu/metrics/request_countwill never receive a message published to
EU/metrics/request_count(note the upper-cased “EU”).
- Avoid overly long topic names (the MQTT specification supports up to 65K bytes). Long topic names will increase your payload size and the cost of message processing on both publishers and subscribers.
- Avoid a leading forward slash when naming topics.
/us/metrics/transactions_processedis a different topic from
us/metrics/transactions_processed. The leading slash is unnecessary.
The MQTT standard that Cloudflare Pub/Sub is built on defines a message as the payload within a . Payloads themselves can be UTF-8 strings, which is what most developers are used to dealing with, or a stream of bytes (a “byte array”) for more complex use cases or for cases where you are using other serialized data formats, such as Protobuf or MessagePack.
In Pub/Sub, which is based on MQTT v5.0, you can also set additional fields to indicate whether the payload is a string or a stream of bytes. Both the
payloadFormatIndicator (0 for bytes; 1 for strings) property and the
contentType property, which accepts a , can be used by the client to more clearly define the payload format.
As a set of best practices when sending Pub/Sub messages, you should consider:
- Keep messages reasonably sized. Buffering data on the client up to 1 KB (or every 2-3 seconds) is a good way to optimize for message size, throughput, and overall system latency.
- Set the
payloadFormatIndicatorproperty when publishing a message. This gives your subscribers or Workers a hint about how to parse the message.
- Set the
contentTypeproperty to the MIME type of the payload. For example,
application/x-msgpackas an additional hint, especially if clients are actively publishing messages in different formats.