# Rate metrics jump after restart: counter resets and rate before sum

Diagnose rate metrics jump after restart: counter resets and rate before sum. Evidence to collect, a worked operational example and recovery checks.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/prometheus-counter-reset-rate/
Language: en

A counter can reset when its process restarts, so raw differences are not automatically rates. Apply PromQL rate to individual series before sum. Otherwise another replica's growth can hide one replica's reset inside the aggregate.

## Confirm the metric type

Preserve the name, exposition HELP/TYPE and measured meaning. Counters accumulate events; gauges can rise and fall freely. Rate is intended for counters. Applying it to a gauge can misinterpret an ordinary decrease as reset. A total suffix helps but does not replace checking exporter implementation. Confirm that labels identify independent instances, since merging multiple sources into one series breaks its meaning before PromQL runs.


[Official reference 1](https://prometheus.io/docs/prometheus/latest/querying/functions/).

## Preserve the restart boundary

Compare raw samples with process restart time and instance identity. With unchanged labels, a replacement process may start from zero. Changed labels create another series while the old one follows staleness behavior. These cases look different. Rate adjusts for breaks in monotonicity and extrapolates within its window, so one value should not be treated as an exact event-by-event history between scrapes.


[Official reference 2](https://prometheus.io/docs/practices/instrumentation/).

## Examine two replicas

Suppose one replica's counter falls from 100 to 0 after restart while another continues increasing. Summing first can partially hide the first reset behind the second replica's growth. Sum(rate(counter[5m])) analyzes each original series before aggregation. The five-minute window illustrates syntax rather than prescribing a universal setting. Choose the window based on scrape frequency and the observation timescale.

| Check | Purpose |
| --- | --- |
| Raw counter | Inspect original samples |
| Resets within the window | Detect individual decreases |
| Per-instance rate | Preserve reset boundaries |
| Sum after rate | Produce an interpretable total rate |

## Correct the expression selectively

Inspect the unaggregated query first, retaining needed labels. Then sum over the intended scope. Do not remove instance identity before rate or replace the counter with an arbitrary absolute difference. Frequent restarts require separate investigation: a corrected formula does not repair a crash loop. Missing scrapes and short windows affect results too. Keep the original expression so dashboard changes remain explainable without claiming the earlier values were exact.

## Test a controlled restart

Generate a known event stream on a test exporter, restart one replica and compare raw values, resets and both query forms. Preserve expected interpretation rather than invented benchmark results. Repeat after changing label policy. This concerns operational rates and observability, not financial usage accounting. Prometheus counters and rate interpolation do not independently define contractual token accounting. Maintain that boundary when using the graph in reports, especially where a smooth rate curve might be mistaken for a complete transactional record.

Related operational reading: [gpu monitoring scorecard](/journal/gpu-monitoring-scorecard/); [model update canary](/journal/model-update-canary/).