In this article
Unique request IDs are unsuitable metric labels. Bounded dimensions and separate trace correlation preserve observability without exploding series counts.
Adding request IDs as labels seems convenient for locating every slow request through metrics. But each identifier creates new series combinations. Combined with model, replica, outcome and histogram buckets, storage can grow faster than dashboard usefulness. Providers need a cardinality budget separating statistical cohorts from unique diagnostic records and accounting for a changing model catalog.
State each label’s question
Prometheus naming guidance promotes meaningful units and labels while avoiding unbounded dimensions. Ask what comparison a field enables before adding it. If the answer is locating one request, a log or trace is probably more appropriate.
Model can be a useful bounded category, but dynamic aliases still need governance. Full prompts, URLs with unique parameters and random request IDs are not safe labels merely because a library accepts strings. Their value spaces differ from status or mode.
Calculate combination products
Estimate model values, replicas and outcomes, then account for histogram representation. Classic histograms add bucket series alongside sum and count. This estimates metric structure rather than useful traffic. Even a rare event can create a new combination.
Keep allowed-value and expected-growth tables. Recheck budgets when models or versions change. Removed replicas can leave historical series, so the current process list does not describe all data retained over the storage period.
Separate counters from diagnostics
Instrumentation guidance connects metrics with service type and observed work. Queries, failures, latency and in-progress work support summaries. Keep definitions stable so cardinality changes do not silently alter measurement meaning.
Unique details belong in permitted diagnostic records with access controls. OpenTelemetry context propagation links operations through trace context without putting a trace ID into every metric observation’s labels. Correlation also does not authorize retention of user content.
Test unknown values
A fixture introduces an unexpected outcome or model ID. Define whether it maps to a bounded unknown value, drops with explicit accounting or follows another documented policy. Silently losing observations can improve a chart exactly when a new failure appears.
Also test that unique IDs do not expand labels. Generate a bounded set of distinct request identifiers and compare series counts before and afterward. Inspect real exporter output because another instrumentation layer may add dimensions.
Version the budget
Record label allowlists, ownership, metric types and migration rules. Repeat fixtures after exporter changes. When the budget is exceeded, investigate new combinations before removing dimensions needed for correct denominators or model separation.
Related guides cover GPU monitoring and operation reporting. The deliverable is a controlled statistical schema with a separate detail path, not free observability or an existing monitoring-system claim. Retain the series-count fixture with the allowed-value table so a later SDK upgrade cannot introduce high-cardinality attributes unnoticed while application code itself remains unchanged.