Key server metrics

The gokeyless key server exposes a Prometheus ↗ metrics endpoint that you can use to monitor signing performance, error rates, connection health, and certificate expiry. This endpoint can also be scraped by the OpenTelemetry Collector Prometheus receiver, making the metrics available to any OpenTelemetry-compatible backend.

By default, metrics are served at:

http://<host>:2406/metrics

The port is configurable via the metrics_port key in your configuration file, the --metrics-port flag, or the KEYLESS_METRICS_PORT environment variable.

The endpoint serves only /metrics. There are no additional HTTP endpoints such as /health or /debug.

All histogram metrics share the same bucket configuration: 15 exponential buckets starting at 100 microseconds, doubling each step up to approximately 1.64 seconds, plus a final +Inf bucket.

Type: Counter
Labels: opcode

Counts every incoming request received over an established connection, regardless of outcome. Incremented once per request before any processing begins.

The opcode label uses the full constant name from the gokeyless protocol.

Type: Histogram
Labels: type, error

Measures the time to execute a single operation, from when processing begins to when a response is produced. For operations backed by a PKCS#11 HSM, this includes the full time waiting for a session from the pool plus the HSM cryptographic operation time.

This metric does not include time a request spends waiting for a connection semaphore slot. That is captured by keyless_request_total_duration_per_opcode.

Opcodes are grouped into coarser categories for this label:

For successful requests the value is no error. All other values indicate a failed operation.

Type: Histogram
Labels: type, error (same values as keyless_request_exec_duration_per_opcode)

Measures the total time to satisfy a request, from when the request packet is read off the wire to when the response bytes are written back to the client.

total_duration = exec_duration + response_write_time

Both timestamps are captured after the connection semaphore is already held, so semaphore queue wait time is not included in either histogram. Under normal load, total duration and exec duration are approximately equal. A growing gap between them indicates slow writes back to the client — for example, network backpressure between the key server and the Cloudflare edge.

Type: Histogram
Labels: None

Measures the time taken by the keystore to locate and return the private key for each request, keyed by SKI, SNI, and server IP.

• For file-backed keystores, this is a map lookup and is typically sub-millisecond.
• For PKCS#11 or HSM keystores, this may include a network round-trip to the HSM if key references are not cached in memory.

This metric is recorded for all signing and decryption operations: OpRSADecrypt, all OpRSASign*, all OpRSAPSSSign*, all OpECDSASign*, and OpEd25519Sign.

It is not recorded for OpPing, OpSeal, OpUnseal, OpRPC, or OpCustom, which do not require a private key lookup.

Type: Counter
Labels: None

Counts connection-level transport failures. This metric reflects problems at the network or TLS layer — it does not count signing errors or key lookup failures, which are reported in the error label of the duration histograms.

Type: Gauge
Labels: source, serial_no, cn, hostnames, ca, server, client

Reports the expiration time (NotAfter) of each certificate loaded by the key server as a Unix timestamp. One time series is emitted per certificate.

This metric is updated:

• At startup, for the server authentication certificate (auth_cert) and the Cloudflare CA certificate (cloudflare_ca_cert).
• On each successful inbound TLS connection, for the peer certificates presented by the connecting client.

sum by (opcode) (rate(keyless_requests[1m]))

sum by (error) (
  rate(keyless_request_exec_duration_per_opcode_count{error!="no error"}[5m])
)

histogram_quantile(
  0.99,
  rate(keyless_request_exec_duration_per_opcode_bucket{type="rsa"}[5m])
)

A value approaching 10 seconds indicates PKCS#11 session pool exhaustion. Refer to Scaling and benchmarking and your HSM documentation for guidance on increasing the session pool size.

histogram_quantile(0.99, rate(keyless_key_load_duration_bucket[5m]))

A spike here without a corresponding spike in exec duration suggests the keystore lookup itself is slow — a possible disk I/O issue or PKCS#11 object enumeration delay.

rate(keyless_failed_connection_total[5m])

A sustained non-zero rate indicates network or TLS problems between the Cloudflare network and your key server.

(certificate_expiration_timestamp_seconds - time()) / 86400 < 30