Separate Triton queue, input, inference and output latency

Diagnose Triton latency through queueing, input processing, inference, and output processing. These intervals have different boundaries, and cumulative counters need interval-based interpretation. One response-time value cannot identify the responsible stage. Correlate metrics with a request trace, especially inside ensembles, to distinguish scheduler waiting from preparation or GPU computation rather than explaining all TTFT through GPU utilization.

Interpret counters correctly

The metrics guide defines cumulative request, queue, and compute-stage durations. An accumulated value is not the latency of one recent request. An interval mean requires matching counter changes and operation counts, while a mean still hides tails. Preserve time range and model/version scope. Mixed versions or counter resets can distort interpretation, so check labels and continuity before calculating. A visually rising graph can simply reflect more work rather than slower requests. State the denominator and interval with every derived quantity.

Trace a particular request

The tracing documentation defines timestamps, request identity, and parent relationships. Use them to inspect queue and compute boundaries for one case. Do not subtract arbitrary client and server timestamps without checking clocks and event definitions. Label unavailable intervals as unobserved. A large unexplained remainder does not prove that all its time belongs to networking or tokenization. Build a timeline identifying each timestamp’s source before making a causal claim about the location of waiting.

Create a stage table

For one slow request, make frontend, queue, compute-input, infer, and compute-output rows. Record observed start/end points, source, and interpretation confidence. This is an artifact template without invented millisecond values. For ensembles, add model and parent identity to avoid summing overlapping spans as strictly sequential work. Compare with a short control request. If input processing grows while infer stays comparable, inspect preparation before replacing GPUs or changing weight precision. The table should guide the next measurement, not just restate total latency.

Respect accounting differences

The documentation describes different compute accounting for cached and normally executed requests. Do not compare differing cache conditions as identical work. Request counts and model executions can also differ under batching. Choose a denominator matching the question. Large cumulative infer time may indicate substantial useful work rather than slower operations. Confirm a suspected regression through a distribution or comparable per-request observation and retain workload conditions. Otherwise a traffic increase can be incorrectly reported as a compute-performance loss.

Report a localized conclusion

Keep interval, labels, counter deltas, sample trace, and batching configuration. Distinguish direct evidence from questions requiring lower-level profiling. An infer-stage bottleneck does not identify a particular kernel, while a queue problem may not benefit from faster tokenization. Repeat the same series after changes at both aggregate and request levels. This reveals waiting moved between stages rather than declaring success when one metric improves but the complete client response remains equally slow. The useful result links latency to a specific stage and a justified next check.

GPU monitoring without excess charts: five observation levels; Throughput and latency: choosing a GPU operating point.