# Planning a GPU pool: when is the next server actually needed?

How observations can guide LLM inference expansion: workload profiles, operating capacity, reserve, site limitations and criteria for the next investment.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/inference-capacity-planning/
Language: en

Counting cards is an easy way to assess a growing pool, but it does not explain useful service capacity. Another server may resolve queues, add reserve or be unnecessary for the current hypothesis. Owners and funding partners should see which limitation each resource removes.

## Define the working unit of capacity

Choose a clear profile: model, weight format, input/output lengths, quality and latency limits. Measure the traffic one configuration handles under those conditions.

Do not use a peak result from one short test as a universal standard. For several request classes, save their proportions and whether those come from observation or assumptions.

[vLLM benchmarking tools](https://docs.vllm.ai/en/stable/benchmarking/) provide a measurement foundation. Deciding which profile is useful for the product remains a separate team and partner task.

## Check demand over time

Daily averages hide peaks and long gaps. Divide observations into agreed intervals and find queue formation. Separate a one-off experiment from a recurring mode.

Without project history, build several synthetic scenarios and limit their testing budget. Do not present forecasts as existing customer traffic.

An idle-server owner may offer a testable role during relevant intervals. If schedule and conditions fit, this can explore a hypothesis before a new purchase.

## Compare expansion with configuration

Before buying a node, consider different concurrency limits, separating short and long tasks, changing model profiles or distributing existing machines more appropriately.

For each option, create a short record: change, expected result, success check and extra costs. Funding discussion then becomes concrete.

If networking or slow recovery is the bottleneck, another card may not resolve it. Measurements, rather than ease of buying a familiar component, should justify the decision.

## Add reserve and site constraints

System capacity is not the sum of every server’s best result. Include maintenance, unavailable nodes and compatibility of selected roles. Reserve requirements depend on the product.

Check power, cooling, space and support. A GPU may need resources absent from its purchase price. Partners should see costs before the next step rather than after delivery.

Define plan review. If workload changes, update the purchasing rationale and service limits together.

## Record the next-step decision

A useful conclusion includes configuration, test budget, observation duration and expansion or stopping conditions. It need not promise exact sales to be persuasive: a transparent link between resource and testable task is enough.

Prepare with [demand validation](/journal/model-demand-validation/), [inference economics](/journal/inference-economics/) and [payment models](/journal/revenue-share/).