# GPU Xid errors: reconstructing an incident timeline

Build an actionable GPU Xid incident timeline from device identity, original logs and workload context, while separating observed sequence from a root-cause hypothesis.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/gpu-xid-timeline/
Language: en

## A code belongs inside a sequence

An Xid message is a useful entry point into a GPU incident, but a screenshot of its number is a weak incident record. The first reported device problem, later consequences and recovery messages may belong to one sequence. Reading only the final line can reverse the apparent cause and effect of the failure.

NVIDIA's guide to [working with Xid errors](https://docs.nvidia.com/deploy/xid-errors/latest/working-with-xid-errors.html) provides a troubleshooting process. Apply the guidance for the actual event and software stack. Do not turn an isolated code into a universal diagnosis that a board, application or driver must be defective.

## Preserve identity and time

Collect a bounded window before and after the event. Retain the host boot identity, GPU UUID when available, PCI address, driver version and the original kernel messages. Index zero is not a durable device identity after hardware changes or a reboot. A fleet report must also distinguish identically numbered devices on different hosts.

Keep the original timestamp representation and the collector's UTC mapping. Kernel-relative times and application wall clocks should not be silently subtracted from one another. When clock alignment is uncertain, show an uncertainty interval instead of reporting an invented millisecond ordering between events from different processes.

## Classify each event's role

Create a timeline with columns for observation time, source, device, message and interpretation. Mark interpretation separately from the original text. The [Xid catalog analysis guide](https://docs.nvidia.com/deploy/xid-errors/latest/analyzing-xid-catalog.html) supplies code-specific context; its recommended actions depend on the event and available evidence, not on how alarming a dashboard color looks.

An illustrative timeline might contain an application timeout, an Xid message and a worker restart. That order alone does not prove that the timeout damaged the device or that the restart repaired it. The next question is whether they refer to the same GPU and whether earlier messages were retained.

## Investigate competing explanations

Compare affected and unaffected workers using the same workload revision. Check whether the event coincided with a host change, a deployment, a transport problem or a recovery action. Preserve a working comparison rather than changing several variables at once. An unavailable GPU query after the event should be recorded as unavailable, not as a healthy zero.

Where a repeatable test is appropriate, define its stop conditions and retain the original incident record separately. A later successful run narrows the observed behavior; it does not erase the earlier failure. Avoid extending a bounded investigation into an untracked stress test on an occupied service.

## Produce a useful incident handover

The final artifact should contain a short event sequence, raw log window, software and device identity, relevant workload conditions, recovery actions and unresolved questions. Distinguish confirmed observations from hypotheses. This makes a support investigation reproducible without claiming a root cause that the available evidence cannot establish.

## Related reading

- [A Change Register for GPU Drivers and Server Settings](/journal/gpu-maintenance-change-register/)
- [Keep a Network Outage Log for a GPU Node](/journal/gpu-owner-network-outage-log/)