Expert parallel vs tensor parallel for MoE inference

Mixture-of-experts models route work through a subset of experts for each token. Expert parallelism places expert ownership across ranks. Tensor parallelism splits tensor computation across ranks. These strategies can interact, but they are not alternative names for using several GPUs, and a model’s active parameter count does not describe its full placement requirement.

Separate stored experts from active work

Inventory all weights that the deployment must make available, then the experts activated for a token. A small active subset can reduce arithmetic without making the remaining expert weights disappear. Attention and shared layers also retain their own placement and memory requirements.

The vLLM parallelism guide distinguishes parallel strategies. Start by mapping the model’s layers and the hardware topology. Asking only whether the active parameter count fits one GPU can produce a configuration that cannot load the model or exchange expert work efficiently.

Draw attention and expert groups

The vLLM expert parallel deployment guide describes interactions among expert, tensor and data parallel groups. Its group relationships apply to the documented configuration, not to every framework implementing MoE.

Draw an attention operation and a routed expert operation separately. Label which ranks share attention work and which own the selected experts. Record replicated and sharded weights explicitly. This prevents a misleading diagram in which every GPU appears to hold the same independent full-model server when it actually participates in a coupled group.

Trace token dispatch and return

A token may need to reach an expert owner and return its result before the next computation proceeds. The communication pattern therefore matters alongside raw GPU compute. Uneven expert selection can concentrate work on a subset of ranks even when average utilization looks acceptable.

NCCL’s communication overview explains available primitives, but the engine’s chosen implementation and fabric determine the actual path. Measure dispatch, expert execution and combination time rather than inferring an ideal all-to-all result from an adapter specification.

Compare topologies under task diversity

Use prompts representing the intended languages, coding tasks and output lengths. Keep the model revision and generation configuration constant while changing the supported group layout. Include bursts and a sustained mixed request stream.

Record per-rank memory, queueing, communication time, completed requests and token pacing. Inspect the slowest rank rather than averaging all GPUs into one attractive utilization number. A topology that fits more experts may still provide worse user latency when dispatch crosses a constrained link too frequently.

Define a qualified execution group

Choose a layout that fits the complete model and meets service objectives under realistic routing imbalance. Document the engine release, group sizes, connector choices and measured links. Treat changes to the MoE backend as changes that can alter both execution and memory requirements.

For a distributed provider, keep tightly coupled expert groups within an explicitly qualified topology. Geographically independent GPUs can serve independent model replicas where appropriate; they should not be advertised as one efficient expert-parallel machine merely because their combined memory exceeds the checkpoint size.

Related reading

vLLM tensor-parallel-size: When One Model Needs Multiple GPUs, LLM memory bandwidth: why gigabytes are not enough.