# A GPU topology matrix: device proximity and actual measurements

Read GPU topology as a device-path map, then test the relevant transfer. Preserve the legend, identities and route used instead of treating distance as bandwidth.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/gpu-topology-distance-matrix/
Language: en

## A topology label describes a path

A GPU topology matrix is a map of relationships between devices. It can help explain which transfers may cross a switch, a host bridge or a more distant system path. It is not a table of measured gigabytes per second, and it does not establish that an application actually used the path that looks shortest.

The [DCGM topology and links guide](https://docs.nvidia.com/datacenter/dcgm/latest/learn/core-services/topology-and-links.html) separates topology information from link state and health observations. Keep those categories distinct. A favorable relationship in an inventory cannot replace a transfer test or prove continuous link health throughout a long inference run.

## Preserve the matrix legend

Save the complete output, its legend, device identifiers and tool version. Do not copy only the cells into a spreadsheet while dropping the explanation of the codes. Preserve GPU UUID and PCI address alongside display indices because enumeration changes can otherwise make two snapshots appear to describe the same pair when they do not.

The [Linux PCI sysfs documentation](https://docs.kernel.org/PCI/sysfs-pci.html) provides device-path information useful for connecting that matrix to the host inventory. Use the platform's documented arrangement when interpreting it. A path inferred from a screenshot is weaker evidence than one tied to actual endpoint identities and the server's installed configuration.

## Form a testable question

Choose the operation being investigated: host-to-device transfer, device-to-host transfer or a particular device pair. A collective communication workload can behave differently from a single transfer. Keep direction, buffer size, concurrency and memory placement constant when comparing paths, and record whether the application supports the relevant direct-transfer mechanism.

An illustrative matrix might show two close devices that nevertheless perform poorly under a simultaneous workload. This does not invalidate the topology observation. It suggests examining shared resources, software path selection or contention. Conversely, a more distant pair may be entirely adequate when communication is a small part of the real workload.

## Avoid turning distance into capacity

Do not sum VRAM because two GPUs are adjacent in the matrix. Model partitioning and engine support determine how their memories can contribute to one workload. A relationship label also does not reserve bandwidth or exclude traffic from other devices sharing parts of the path.

Record a capability check and a measured result separately. If one pair is unsupported, leave it unsupported rather than inserting a zero-speed observation from a failed test. A test that fell back to a host-mediated route needs that route documented so it is not compared as if it exercised the same mechanism as another pair.

## Keep an explanatory artifact

Publish the inventory matrix beside a measurement table and a short description of the tested operation. Include operating conditions and unexpected results. This lets another engineer reproduce the question rather than merely admire a topology diagram. The useful conclusion is which path mattered to the workload, under which conditions, and what the matrix alone could not establish.

After changing the server configuration, retain a new matrix version. Comparisons should not silently combine different paths after moving a board, replacing a riser or changing the devices available to the host.

## Related reading

- [NUMA in GPU servers: why resource placement matters](/journal/numa-and-gpu-servers/)
- [Multiple GPUs in a pool: one service or one large model?](/journal/multi-gpu/)