# Head vs tail sampling: why a trace splits across Collectors

Diagnose head vs tail sampling: why a trace splits across Collectors. Evidence to collect, a worked operational example and recovery checks.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/otel-head-tail-sampling-collector-routing/
Language: en

Tail sampling needs spans from one trace to reach the same sampler instance. If parts go to separate Collectors, decisions can use incomplete information. Head sampling decides earlier, and a later tail stage cannot recover spans discarded upstream.

## Locate every sampling stage

Record the SDK sampler, intermediate processors and backend policy. Mark where data is dropped and which routing keys apply at each transition. A missing span may never leave the SDK, be lost from a queue or miss a tail decision. These are separate causes. Increasing decision_wait cannot reunite trace fragments that are routed to independent processes and never assembled together.


[Official reference 1](https://opentelemetry.io/docs/concepts/sampling/).

## Separate head and tail decisions

Head sampling decides early without full knowledge of future spans. Tail sampling considers accumulated spans and can select by error or duration, but requires state and memory. They can be combined, although the tail sampler only sees data that reaches it. An upstream head drop cannot be undone by a downstream “keep errors” policy. Preserve that boundary when explaining an absent error trace.


[Official reference 2](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor).

## Examine two Collectors

In a test, spans sharing one trace_id are distributed round-robin between two tail-sampling Collectors. One sees the beginning, the other an error, and both decide from incomplete data. Introduce trace-aware routing before the tail stage, for example using the appropriate load-balancing exporter. All spans of one trace must reach one sampler. Backend membership changes can remap routes, so test scale-out as well as stable membership.

| Check | Question answered |
| --- | --- |
| SDK sampling | Which spans were exported at all? |
| Routing key | Does one backend receive the trace? |
| Decision timing | Did relevant spans arrive in time? |
| Cache and capacity | How are late data handled? |


[Official reference 3](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/loadbalancingexporter).

## Inspect waiting and capacity

Decision_wait controls timing according to processor version and sampling strategy. Follow the installed version's documentation, especially when newer modes exist. Num_traces bounds retained state; late-span behavior depends on decision cache and prior decisions. More waiting increases retained-data requirements and is not a universal loss prevention measure. Do not copy current-main defaults into an older deployed build without checking supported options.

## Verify a complete test trace

Generate a synthetic trace with known spans and delays, then check SDK export, selected Collector and the final set. Repeat with a late span and changed sampler membership in a test environment. Preserve topology, routing key, versions, policy and expected decisions. Success establishes correct decisions for the tested flow rather than completeness of arbitrary historical traces. Sensitive prompts are unnecessary; a known synthetic structure is sufficient. This also makes the test repeatable without depending on real customer traffic or changing application propagation contracts.

Related operational reading: [gpu monitoring scorecard](/journal/gpu-monitoring-scorecard/); [hardware security](/journal/hardware-security/).