In this article
An NVLink delta requires the same field, scope and epoch; an absent value must not be treated as zero.
Start with generation and topology
NVLink telemetry depends on hardware generation and tool release. A universal parser expecting identical lines for every GPU can silently lose data after an update. The dcgmi nvlink documentation distinguishes output organization by generation: Hopper and older devices expose link sections, while Blackwell and newer devices expose GPU level fields. Record model, architecture, DCGM and driver versions before collecting available link states. A link index alone is not a sufficient identifier across machines. Keep it associated with the device and topology snapshot so comparisons concern the same physical path.
Preserve each field’s meaning
The DCGM topology and links guide helps connect observations to system connections. A metric register should record the name, field identifier, scope, units and value type. Cumulative errors, current link state and BER must not share one processing formula. Computing deltas from an already normalized measurement merely because it appears beside counters is particularly dangerous. Preserve unsupported or blank values as unknown with a reason. Documentation specifically notes that missing Blackwell values do not mean zero. Such gaps must remain visible in charts and diagnostic records instead of being hidden behind a green success line.
Calculate deltas within one epoch
For a cumulative counter, record values before and after a bounded interval, together with time and device identity. If a value decreases, do not publish negative errors: investigate resets, restarts, GPU replacement and data source changes first. A new epoch gets a new baseline. Historical starting values must not be presented as errors from the current experiment either. Preserve interval duration and workload composition when comparing windows, since equal deltas under different exchange volumes describe different conditions. No universal alarm threshold follows without the model, error type and manufacturer guidance; classification depends on the documented meaning of the field and its observation context.
Correlate with the workload
A diagnostic experiment compares a quiet window with bounded inter-GPU exchange under a known configuration. Preserve link state, operation duration and system events simultaneously. Growth during exchange justifies investigating the path, rather than automatically proving a faulty GPU. If topology or field scope changes, correct the data association first. Do not simply add GPU level and per-link values: scopes may overlap and meanings may differ. Keep resets outside ordinary observation unless separately justified, since they may erase useful history and complicate reproducibility of the symptom in a subsequent window.
Prepare a diagnostic package
The package contains raw tool output, versions, topology snapshot, baseline, final values and reasons for time series breaks. See multi-GPU configurations and vLLM tensor parallelism. Limit the report’s claim to an observed delta for a specified field under selected work. No increase does not establish perfect interconnect health, especially when some telemetry is unavailable. A positive delta does not establish its cause without additional investigation. This presentation preserves technical accuracy across hardware generations and helps distinguish actual system changes from output format changes, incomplete support or incorrect aggregation of different counter scopes.