In this article
PCIe is part of the data path, not a standalone prediction of model speed. Measure topology and communication for the application.
Identical GPU models can occupy different slots and server platforms. Some workloads then behave differently because of connections to the CPU, network and other accelerators. When buying a card or offering a machine to a pool, examine the data path as well as the accelerator specification.
Start with topology
Record installed devices and how they connect. Interface generation and available lanes describe a path, while actual behavior also depends on platform and configuration.
NVIDIA CUDA Best Practices treats host-device transfers as a separate optimization concern. NCCL troubleshooting identifies GPU connections and topology for targeted investigation. Do not turn theoretical interface bandwidth directly into predicted tokens per second: the application sits between those measurements.
Separate loading from steady operation
A model may transfer substantial data during startup and follow another pattern afterwards. An improvement limited to loading can help recovery without accelerating each generation. In an illustrative pilot, measure startup from locally available weights, the first request and a steady warmed-up stream. Preserve context limits and environment versions.
Test CPU-GPU offloading separately when data moves between RAM and VRAM. It differs from execution entirely within accelerator memory, so one timing result should not represent both placement schemes. Record which phase the proposed connection improvement is meant to address.
Check multiple GPUs
Devices participating in one distributed model exchange data differently from independent replicas. Total VRAM alone therefore does not justify an architecture. Where supported, compare independent operation and an agreed distributed configuration. Define the intended improvement: model placement, throughput or another criterion.
If communication limits the result, another GPU may not provide the expected effect. A different role for existing devices may be more useful, but that conclusion still needs quality and operating-cost checks. Preserve the actual model and workload conditions rather than treating device activity as useful completed work.
Include storage and networking
The GPU does not operate in isolation. Data loading, network traffic and storage activity can share platform resources. Test concurrent operation representative of the owner’s normal work, or an agreed synthetic equivalent. Compare queueing, latency and errors. A problem appearing only in that mode differs from a defective card.
Do not change system protections or hardware randomly to improve a number. Begin with recorded observations and an agreed diagnostic plan that can be reversed. Before moving a card, arrange downtime and rollback. Afterwards, repeat the full application set: improving one transfer path does not guarantee preservation of other required connections or resources.
Describe the configuration to partners
The final diagram should show GPUs, host, storage and networking with tested limits. That supports a funding decision better than an unexplained request to buy another card.
Read about NUMA, multiple GPUs and benchmark methods.