# Data parallel vs tensor parallel: replicas or a split model?

Choose between independent serving replicas and tensor-sharded execution using model fit, cache locality, failure boundaries and measured communication.

Author: Retrivium team
Updated: 2026-11-03
Canonical URL: https://retrivium.ai/journal/data-parallel-vs-tensor-parallel-serving/
Language: en

Adding GPUs to a serving deployment can mean creating additional request-serving replicas or splitting one model's work across devices. Data parallel serving and tensor parallel execution address these different goals. They can also be combined: an independently scheduled replica may itself contain a tensor-parallel group. Define the unit of execution before comparing capacity.

## Check whether the whole model fits

Estimate weights, runtime workspaces and active-request state for the intended context and concurrency. A model fitting in memory at startup is not the same as a useful serving configuration. Keep headroom for its actual request mix.

The [vLLM parallelism guide](https://docs.vllm.ai/en/stable/serving/parallelism_scaling/) connects placement to model fit and hardware. Tensor parallelism can make a model executable across devices, but it also introduces communication. When a complete useful replica fits on one device, replication becomes a separate candidate rather than an automatic reason to shard it.

## Define the replica group

The [vLLM data parallel documentation](https://docs.vllm.ai/en/latest/serving/data_parallel_deployment/) describes serving configurations with multiple data-parallel ranks and interactions with other layouts. Avoid assuming that every process is a fully independent single-GPU copy: the model architecture and configuration determine the group.

Draw the request assignment boundary. Mark where one request remains until completion and which ranks jointly execute it. For a hybrid layout, state both the number of independent serving groups and the devices per group. This makes hardware accounting and restart scope understandable.

## Include cache locality in replica scaling

More replicas can spread incoming work, but they can also spread repeated prefixes across separate caches. The [prefix caching guide](https://docs.vllm.ai/en/latest/features/automatic_prefix_caching/) explains why the presence of matching state matters. Capacity gained from replication should not be estimated from a benchmark that warms only one worker.

Replay conversational sessions as well as independent requests. Compare ordinary balancing with the intended session or cache-aware policy. Measure whether one popular context overloads a replica while others sit idle. Replication changes placement opportunities, not just the number of available GPU counters.

## Compare equal hardware budgets

Keep total devices, model revision, prompt trace and generation settings visible. Compare supported layouts using useful completions, first-token tails, token pacing and per-device memory. Include communication time for tensor groups and imbalance for replicas.

Also test losing one worker. A failed rank inside a coupled execution group can have a different service impact from a failed independent replica. Count unavailable model capacity and recovery time rather than simply counting processes that remain running after the failure.

## Choose the bottleneck you need to solve

Use a split-model layout when fit or measured execution behavior justifies the communication. Use additional independent groups when the goal is serving more concurrent eligible requests and the complete model can be provisioned accordingly. Neither strategy guarantees linear throughput scaling.

For multiple sites, define which locations host self-contained replicas and which devices participate in one tightly coupled group. Combining the memory of distant machines on a spreadsheet is not a demonstrated tensor-parallel configuration. The placement decision should match the actual network and failure ownership of the provider's infrastructure.

## Related reading

[vLLM tensor-parallel-size: When One Model Needs Multiple GPUs](/journal/vllm-tensor-parallel-size/), [Prefix caching: when repeated context helps a GPU pool](/journal/prefix-caching/).