# PCIe AER near a GPU: severity, requester and chronology

Read a PCIe AER report near a GPU through severity, requester identity, and reporting topology.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/pcie-aer-gpu-evidence/
Language: en

Read a PCIe AER report near a GPU through severity, requester identity, and reporting topology. The component printing a message is not necessarily the root cause. Preserve full logs and chronology before resets or reboots, then correlate GPU Xids and application failures. One line naming a GPU or Root Port cannot establish which card, cable, slot, or driver caused the incident.

## Classify severity first

The [Linux AER guide](https://docs.kernel.org/PCI/pcieaer-howto.html) distinguishes correctable and uncorrectable events, with non-fatal and fatal categories among the latter. Their interface and recovery implications differ. Do not label every report a critical GPU failure, but retain repeated corrected reports for analysis. Preserve classification, status/mask, and available transaction information. Message rate limiting also means log-line count need not equal the number of all underlying hardware events. Keep that limitation visible when interpreting incident frequency instead of presenting a filtered console as a complete event counter.

## Map requester and reporting path

AER reports travel through a PCIe hierarchy. Preserve BDF identities and topology, including endpoint, switches where present, and root port. Requester identity has a defined reporting meaning; it does not replace root-cause investigation. Do not blame the address appearing at the beginning of a line automatically. Several devices behind a common upstream component may react together. Their shared behavior calls for examining the branch, not only the card first noticed by the application. Keep reporting location separate from a confirmed failed component.

## Build a timeline record

For an illustrative incident, create rows for AER, GPU Xid, application-response loss, and observed recovery. Record timestamp, clock source, BDF or GPU identity, and original message. This is a template rather than a report of an actual outage. Order events using an aligned timeline. If source clocks differ, retain uncertainty rather than claiming that a kernel event preceded inference failure by a precise interval and therefore caused it. Temporal proximity alone cannot establish that mechanism.

## Add GPU-specific evidence

The [NVIDIA Xid guide](https://docs.nvidia.com/deploy/xid-errors/latest/working-with-xid-errors.html) describes collecting and investigating GPU errors. Xid and AER can describe one episode without forming a universal one-to-one mapping. Preserve driver version and GPU identity. Missing Xids do not invalidate AER observations, while a present Xid does not independently identify a faulty physical part. Correlate additional evidence and supported platform procedures instead of resetting an active card merely to test a hypothesis. Keep observation and intervention as separately timestamped actions.

## Preserve evidence before intervention

The package should contain full logs, topology, versions, workload phase, and post-event observations. State maintenance changes and their times. Do not disable reporting or reset production devices as an incidental diagnostic step; those actions can erase evidence and affect neighbors. Use an agreed test environment for reproduction. The conclusion must distinguish an interface event, its reporting path, and an established cause. If cause remains unknown, say so rather than assigning blame to a GPU because nearby log messages coincided with a temporarily unresponsive LLM endpoint.

## Related reading

[PCIe and GPU hosting: when the connection limits a model](/journal/gpu-pcie-bandwidth/); [Keep a Network Outage Log for a GPU Node](/journal/gpu-owner-network-outage-log/).