In this article
Compare RDMA over Ethernet with InfiniBand through adapters, switches, congestion and real inference transfers, without treating link speed as an SLO.
RoCE and InfiniBand can both support RDMA-based communication, but they are not interchangeable settings on an arbitrary network. RoCE carries RDMA over Ethernet; InfiniBand has its own fabric architecture. For LLM serving, the useful choice depends on the complete path and communication pattern rather than a nominal adapter speed.
Start with the serving operation
Separate communication among model ranks from KV transfer between workers and ordinary API traffic. A tightly synchronized tensor-parallel operation can respond differently to latency and congestion than a large asynchronous state transfer.
The NCCL overview describes communication across supported interconnects. A library selecting an available transport does not establish that the underlying fabric meets an inference latency objective. First identify which operations are on the critical path of the deployed model.
Compare complete compatible fabrics
The RDMA programming manual explains transport distinctions. Inventory the adapters, ports, firmware, switches, cables, routing and host software for each proposed path. Verify that the intended adapter mode and switch environment actually correspond.
The same programming concepts do not make an Ethernet switch an InfiniBand switch or guarantee identical device support. Record which configuration was qualified by the platform supplier and which parts are an operator-designed integration. Treat the actual physical and software combination as the test subject.
Include congestion and competing traffic
NVIDIA’s RoCE data-center whitepaper discusses the surrounding Ethernet design. Reading it is not a reason to copy congestion or flow-control settings blindly. Such settings affect a whole fabric and must be chosen within its validated design.
Measure serving traffic while other intended workloads are active. Include several senders targeting the same receiver and simultaneous transfers in both directions. A quiet point-to-point test cannot characterize queue buildup or interference at the shared switch and host links.
Measure latency distribution and failure
Use the actual message sizes, rank group and concurrency produced by inference. Report useful payload transfer time separately from setup, synchronization and application queueing. Record tail behavior, retransmission or error signals, and whether communication competes with host-to-device copies.
Then exercise a controlled link or worker failure in an isolated test. Verify timeout and recovery behavior at the serving layer. A fabric that delivers an attractive peak bandwidth can still leave unacceptable stalled requests if the application has no bounded response to a failed peer.
Keep cluster claims separate from wide-area networking
Neither RoCE nor InfiniBand makes geographically distant partner nodes behave like GPUs behind one local switch. Wide-area paths have different latency, routing and operational ownership. A distributed provider can instead place tightly coupled ranks within qualified locations and route independent requests among locations.
Choose a fabric using measured inference requirements, available operational expertise and the entire lifecycle cost. State the supported topology and test conditions in any comparison. Do not turn a protocol name into a universal tokens-per-second guarantee or imply that upgrading network adapters alone fixes a poorly placed parallel model.
Related reading
vLLM tensor-parallel-size: When One Model Needs Multiple GPUs, LLM memory bandwidth: why gigabytes are not enough.