# Open and closed load models for an inference API

Why fixed concurrency can hide inference API overload, and how to compare open arrivals with a closed load model in a bounded experiment.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/provider-open-closed-load-tests/
Language: en

## Describe the source of requests

A load test measures both the server and the way work is generated. In a closed model, a fixed client population starts another request after the previous one completes. In an open model, arrivals are scheduled independently of response duration. The [k6 documentation](https://grafana.com/docs/k6/latest/using-k6/scenarios/concepts/open-vs-closed/) explains this distinction and coordinated omission. When generation slows, a closed test reduces its own arrival rate, sometimes concealing queue growth that independent user demand would create. Both modes are useful, but answer different questions. A virtual user count without its submission loop is therefore insufficient for interpreting latency charts or estimating a throughput boundary.

## Keep useful work comparable

Preserve the model, input length distribution, output limits and response modes across comparisons. Otherwise a changing token workload can be mistaken for an effect of the load model. One experiment sets fixed concurrency; the other uses a bounded arrival schedule. Values are experiment parameters rather than universal infrastructure recommendations. Define maximum duration, request count, total token budget and a queue based stop condition before each run. The test must not create work indefinitely after the service has stopped admitting requests. Retain rejected and unsent jobs separately from successful completions instead of removing them from the report.

## Check generator capacity

An open scenario requires the client machine to keep up with the schedule. If it lacks connections, CPU time or available workers, actual arrivals lag behind the plan. That alone does not establish an inference server limit. Compare scheduled, actually sent, accepted and completed request counts. Measure scheduling delay inside the generator separately. [Prometheus instrumentation guidance](https://prometheus.io/docs/practices/instrumentation/) usefully distinguishes current work, completions and errors. For streaming, record the start and end of body consumption as well: fast headers do not mean that the client can consume the whole stream without building a local memory backlog.

## Compare latency with queue behavior

The diagnostic branch starts with actual arrival rate. If it falls while response duration rises in a closed test, investigate workload self regulation. If an open test maintains arrivals while the queue grows, examine admission, resource limits and rejections. [vLLM metrics](https://docs.vllm.ai/en/stable/usage/metrics/) provide examples of engine side observations, although names and availability depend on version. Looking only at the median of completed responses is dangerous: unfinished requests may never enter that sample. Show active requests remaining at test termination, stopping reasons and rejection fractions alongside latency distributions for the corresponding observation window.

## Bound the conclusions

The experiment artifact contains the schedule, actual arrivals, workload composition, component versions and rules for unfinished work. See the [vLLM bench serve command](/journal/vllm-bench-serve-command/) and [benchmark methodology](/journal/inference-benchmark-method/). A closed model describes a fixed active client population; an open model explores independently arriving jobs. Neither short run automatically predicts commercial traffic, every model's behavior or long term hardware stability. Results remain meaningful only with generator limitations and the examined load range. Repeat an experiment after explaining a discrepancy, rather than averaging incompatible scenarios into a single attractive number.