In this article
A passive incident and a later passing diagnostic can both be valid because their scope and conditions differ.
Two questions require two kinds of evidence
A node can have a clean recent monitoring report and still fail a selected diagnostic workload. It can also retain a historical health incident after a later active test succeeds. Those outcomes are not automatically contradictory. They answer questions about different observation windows, selected components and conditions.
The DCGM diagnostics guide distinguishes active tests from health monitoring during ordinary operation. An active result is evidence from the specific test that ran. It is not a reconstruction of every workload the node previously executed, and it does not repair the underlying fault or establish a warranty decision.
Establish the passive observation window
Record the watched systems, selected entities, sampling interval, retention and host-engine lifetime. The dcgmi health reference explains that checks evaluate retained data for enabled watches. No incident in a report therefore needs the accompanying question: was the relevant condition actually observed and retained?
If monitoring began after the application failed, the report cannot prove that the earlier interval was healthy. Similarly, a short retention window may have discarded an old incident. Keep unavailable and unwatched intervals visible instead of turning every empty incident list into a continuous health guarantee.
Scope an active test deliberately
Choose the entity and diagnostic suite that address the suspected path. Preserve the installed DCGM release, invocation, selected tests, environment and raw result. A software-deployment check and a sustained hardware workload do not establish the same property, even when both finish with a favorable status.
Active diagnostics can consume compute, memory, interconnect, power and cooling resources. Coordinate the test with the scheduler and the platform’s maintenance process. A test run beside an occupied application may measure interference or disturb the application, weakening both the diagnostic evidence and the workload result.
Reconcile a seemingly mixed result
Consider an illustrative case: passive monitoring records a thermal-related incident during an earlier busy period, then a later memory test passes under different conditions. The memory result does not erase the thermal observation. The next investigation should preserve both conditions and examine the path relevant to the original failure.
The reverse case is also informative. An active test may fail while passive monitoring shows no incident because the failing path was not exercised during the retained interval, the relevant watch was absent or the diagnostic encountered an environmental issue. Do not resolve the difference by choosing whichever green or red result seems more convenient.
Create a combined evidence record
Use separate sections for passive coverage and active execution. Include device identity, timestamps, environmental conditions, raw output, failures, skips and unresolved questions. Connect each conclusion to the evidence that supports it. The result is a bounded statement about what was observed and tested, which is more useful for returning a node to service than a single unexplained healthy label.
Retain the reason for the next investigative step. Repeating a passing test without changing the question adds little evidence when the original failure concerned another path, while consuming maintenance resources.