In this article
A missing Prometheus series is not zero. Staleness means a previously observed series stops being returned by the relevant query. Inspect up and target state, then determine whether the entire source disappeared or only a particular metric family.
A missing Prometheus series is not zero. Staleness means a previously observed series stops being returned by the relevant query. Inspect up and target state, then determine whether the entire source disappeared or only a particular metric family.
Preserve the exact query
Record expression, time range, step and selectors. Dashboard transformations may replace missing values and conceal the original Prometheus result. Inspect the untransformed response first. Check whether job or instance changed during deployment: a new labelset creates another series. Querying an old identity can resemble data loss while the exporter publishes under a new label. Record which Prometheus or remote backend answers the query.
Understand staleness and lookback
For instant expressions, Prometheus selects a current sample within the lookback window. Once marked stale, a series stops returning beyond the relevant boundary. Exporters supplying their own timestamps have additional behavior. Avoid explaining every gap with one universal interval without checking version and configuration. Range functions and instant selectors also process their inputs differently. Interpret a missing graph point in the context of the expression rather than the line’s appearance alone.
Compare three cases
In a test, the exporter first becomes unavailable, then its target is removed from discovery, then it returns without one metric family. In the first case, up helps reveal scrape failure. In the second, even up for the removed target eventually stops being current. In the third, up may remain 1 while an application metric is missing. A blanket zero-fill would hide these differences.
| Observation | Possible scope |
|---|---|
| Up is zero | Scraping failed |
| Up is absent | Target removed or selector mismatch |
| Up is one, metric absent | Scrape contents changed |
| Metric explicitly zero | Source reported a value |
Correct absence detection
Use appropriate functions and expressions for missing series with the intended labels and time scope. Keep missing observation distinct from a measured zero. For relabeling changes, compare scraped and post-relabel sample counts. When identity changes, update discovery and dashboards deliberately while preserving historical interpretation. Exporter reachability does not establish completeness of a required metric family.
Verify the returned series
Confirm current samples and labels after correction, then examine the historical gap separately. New points do not automatically backfill past losses. Preserve the expression, target state, up and a safe sample of raw scrape data. Define explicitly how alerts should react to missing observation. This prevents “zero errors” from being reported when measurement itself has stopped returning data. It also keeps service failure distinguishable from observability failure during later incident review, when a visually continuous dashboard might otherwise conceal the original gap.
Related operational reading: gpu monitoring scorecard; network and uptime.