In this article
Handle counter identity, support and reset boundaries before calculating or aggregating a rate.
Every cumulative value has a history
A GPU counter is meaningful only within the history it accumulates. A collector restart, driver lifecycle event, device replacement or supported hardware reset may divide that history. If the monitoring system stitches both sides together without an explicit boundary, its calculated error or event rate can become more misleading than the original readings.
The NVML query reference documents different counter families and device-specific availability. Their lifetimes must be checked individually. Do not assume that every field resets with the collector, or that every cumulative field survives a device or driver transition in the same way.
Define a measurement epoch
An epoch is the interval during which identity, counter semantics and baseline remain comparable. Give it a local identifier and retain the host boot identity, GPU UUID, driver version and collector configuration. A new epoch can be necessary even if the numeric value did not decrease, because a device or reporting scope changed.
The DCGM field reference helps identify the meaning of selected fields. Keep vendor field identifiers and units in the collection configuration. A convenient display name such as errors total should not hide whether the series describes one device, an instance, a particular memory location or a different event category.
Handle a decrease explicitly
Consider hypothetical consecutive values of 40 and 3. Their simple difference is negative 37, but that is not a physical rate of repaired errors. It is a signal to examine the epoch boundary. If a reset is confirmed, begin the new baseline at the new observation. If the transition is unknown, retain the uncertainty.
Do not calculate an exact count across an unobserved restart merely by adding the post-restart value. Events may have happened before the first new sample, and the field’s accumulation lifetime may differ from the host’s. Explain what the observation covers and what remains outside the measured interval.
Preserve missing states
A timeout, unsupported field and valid zero are different records. Keep query status separate from numeric value so that an exporter outage cannot become a row of healthy-looking zeros. Likewise, a changed GPU index should not attach the old device’s history to the new occupant of that index.
For a fleet report, perform reset handling within each original series before aggregation. Otherwise one device’s restart can hide another device’s increment. Keep the aggregate linked to the underlying device epochs so an incident reviewer can reconstruct the contributing observations.
Validate the collector with fixtures
Use saved, non-disruptive input sequences: increasing values, unchanged values, a decrease, a missing sample, an unsupported field and a changed device identity. Check the resulting boundaries and calculated differences. This tests the reporting logic without resetting a live GPU merely to generate a demonstration. The final report should describe the measured epoch, not imply an uninterrupted device lifetime that the collector never observed.