Correlate SGLang request latency across workers with OpenTelemetry

SGLang tracing can explain one request’s latency across workers when aggregate TTFT is insufficient. It requires request identity, correlated timing, and an appropriate detail level. Enabling OpenTelemetry does not itself locate a problem. Define the interval being investigated and verify that the trace covers the relevant processes rather than ending at an upstream network component before computation begins.

Choose a tracing question

For example, investigate where a long request waits before decode. The production request-tracing guide describes tracing controls, OTLP export, and detail levels. Select a level for the question rather than collecting maximum detail without purpose. Verify that the collector receives events and associates the relevant spans. An empty view can indicate an export problem; it does not prove that the server did no work or passed through a stage instantaneously. Validate observation before interpreting an apparent absence.

Correlate identities explicitly

Preserve the client identifier, associated trace, and worker roles available in the selected release. Similar start times are insufficient when many processes handle requests arriving together. Check context propagation and label missing links. Do not join spans manually merely because their durations add up to an expected response time. Under concurrent load, that can create a convincing but incorrect route assembled from several unrelated requests. Keep the correlation method visible in the diagnostic artifact.

Record one slow request

Capture sending time, server stages, first token, and completion. Identify each timestamp’s source: client clock, trace span, or separate log. This defines a record without inventing millisecond values. If a large gap lies between spans, check whether the interval is observable at all. A missing span is not missing work. Compare the operation with a shorter input while preserving model and output policy, then locate the stage whose behavior changed rather than comparing total durations alone.

Add lower-level profiling

The SGLang profiling guide covers CPU/GPU tools for deeper investigation. A request trace selects the interval; a profile explains activity inside it. Do not claim a particular kernel bottleneck when a trace exposes only an overall stage. Profile the necessary workers separately in a distributed deployment. Aggregate GPU utilization without request correlation cannot replace this check and may hide one request’s waiting behind useful computation for others. Retain the relationship between each profile and traced request.

Measure observation overhead

Repeat a small controlled series with tracing enabled and disabled using the same arrival schedule. Assess event volume and latency effects rather than assuming zero cost. Keep export settings, versions, worker layout, and a minimal reproducible request sequence. Synthetic fixtures need identifiers and known inputs, not unnecessary user text. Distinguish directly observed boundaries from inferred gaps in the conclusion. Repeat after tracing-level or runtime changes because additional spans alter visibility without necessarily altering execution. The result should localize a particular wait, not substitute an attractive but incomplete diagram for causal evidence.

GPU monitoring without excess charts: five observation levels; Throughput and latency: choosing a GPU operating point.