TECHNOLOGY

From a slow histogram bucket to a trace through an exemplar

In this article
THE QUICK TAKE

An exemplar provides a linked sample rather than a complete outlier list. A missing trace does not establish that no slow request occurred.

A histogram can show increasing latency without explaining one request. Putting every request ID in labels is expensive and conceptually wrong. Exemplars offer another path: selected observations can link to trace context while the aggregate metric remains bounded. Providers should test that navigation chain and explain sampling so one example is not mistaken for a complete slow-operation journal.

Separate aggregate from sample

The OpenTelemetry metrics data model describes exemplars as observations with additional context. They do not replace histograms or create a series for every request ID. Metrics answer statistical questions, while an exemplar helps locate one example within that measurement’s scope.

Name the linked value: total duration, first useful event or another defined quantity. A chart for one SLI must not link to an arbitrary similarly named span and call that evidence. The observation and trace need an identifiable shared operation.

Check context propagation

OpenTelemetry propagation connects distributed operations. An exemplar requires appropriate context when recording the observation. Measuring outside the intended scope can leave links absent or associated with another operation even when the histogram itself is correct.

Preserve safe trace and span identifiers without prompts or secret headers. Technical-event association is enough. Trace-backend access policy remains separate: seeing aggregate charts need not automatically grant access to every detailed diagnostic record.

Account for sampling and retention

Not every request has a retained trace. Instrumentation sampling, backend policy and storage duration all matter. Missing trace detail therefore does not establish that no slow operation occurred. It may reflect observability limits or mismatched retention periods.

Define interface behavior when detail is unavailable. Showing the limitation is better than opening an unrelated trace. One successful demonstration does not establish that every outlier is retained merely because the backend supports exemplars.

Build a control path

A synthetic fixture creates a known operation with measured duration and an accessible test trace. Check histogram recording, exemplar presence under the selected policy and the identity of the linked operation. Real customer requests are unnecessary.

The Prometheus histogram guide explains distribution interpretation and aggregation pitfalls. Exemplars do not automatically improve percentile precision. They add diagnostic links while bucket resolution, cohorts and units still need correct definitions.

Preserve diagnostic limits

One slow trace may suggest queueing or delivery delays without explaining the entire distribution. Check whether its profile belongs to the cohort under investigation. An unusual request is not proof of a common degradation cause merely because it is easy to open from a dashboard.

Related guides cover GPU monitoring and throughput with latency. The deliverable is a verified link from a selected sample to a known trace, with limitations, without unique IDs in metric labels or complete-infrastructure tracing promises. Keep the missing-trace case in the fixture set so later interface changes remain honest when sampling or retention removes the expected detail.

← All articles