In this article
Averaging replica p95 values does not produce service p95. Aggregate compatible distributions and retain the selected request cohort.
Each replica has a p95 chart, making a simple average tempting. But a percentile depends on the full distribution and observation count. A low-traffic replica should not automatically carry the same weight as a busy one. Inference APIs need compatible histograms and explicit cohorts, or the result looks precise while answering the wrong question.
Define the summary question
Specify the measured duration: full response, first useful content or another SLI. Select model, endpoint, mode and interval. Mixing short and long workloads can be appropriate for a whole-service view but differs from evaluating one task profile.
The Prometheus histogram and summary guide explains why precomputed quantiles cannot be aggregated arbitrarily. Averaging two p95 values cannot reconstruct the combined distribution. A coincidentally plausible result does not validate the method when traffic shares change.
Preserve distributions
Histograms retain observations in a form that supports combination under appropriate conditions. Classic histograms depend on bucket boundaries; other variants require checking tooling support. Similar metric names do not make incompatible series aggregatable.
The vLLM metrics reference documents observations and histogram bucket configuration. Inspect the installed version and actual endpoint output. A documented name does not prove identical types and boundaries exist on every instance after updates.
Choose bucket resolution
Boundaries should provide useful precision around the threshold of interest. Broad buckets give coarse estimates, while excessive detail increases storage cost. Do not promise arbitrary p95 precision without explaining distribution representation.
Create a small synthetic latency set and calculate expected properties independently of the dashboard. Feed it through the selected histogram type. This tests queries and exporters without real generation. Retain raw values so rounding cannot silently explain discrepancies.
Bound cohort labels
Prometheus naming guidance supports consistent units and label meaning. Bounded dimensions such as model or outcome can help summaries. Unique request IDs should not become labels on every observation: they multiply series without fixing percentile aggregation.
Do not merge successes and truncations without policy. If failure durations are separate, dashboard queries must reflect that. Otherwise quick overload rejections can improve overall p95 while users receive fewer useful answers.
Validate queries after releases
Save expressions, histogram type, boundaries and filters with instrumentation version. Repeat the synthetic fixture when names or units change. Lower p95 after a definition change is not evidence of acceleration, even if the chart looks better.
Related guides cover GPU monitoring and benchmark methodology. The result is an explainable percentile for a defined cohort rather than a universal SLA or performance claim about unknown infrastructure. Comparable observations matter more than one convenient dashboard number. Keep request counts beside the percentile as well, since an interval containing very few observations should not be interpreted with the confidence of a sustained, representative workload.