Manage resident models and eviction in the llama.cpp router

Validate the llama.cpp model router through actual residency and child-process state. A preset appearing in a list does not mean its weights are already loaded. Model-count limits and memory margins address different constraints. Map load, idle, and unload transitions for the intended set so that cold starts are not mistaken for API-name failures or generation-speed differences when checkpoints compete for resources.

Distinguish presets from processes

The server documentation describes router presets and model-loading controls. A preset defines configuration; an actual instance has a lifecycle. Even a request with the correct model ID can wait for loading. Check public identity, process presence, and inference readiness separately. Renaming an alias cannot solve insufficient residency capacity. Public identifiers do not determine which weights remain on devices between requests or which process currently serves them. Record these states rather than inferring them from a successful model-list response.

Apply both residency constraints

A loaded-model count constrains membership, while a memory margin reserves free space per device. Equal model counts can require different memory because weights, context, and support buffers differ. Pin settings supported by the build without promising arbitrary checkpoint combinations. The model-management implementation exposes state and reload/unload behavior. Acceptance must account for whether a candidate is idle and eligible for unloading rather than assuming every resident process can be removed while it is actively serving a request.

Exercise three presets

Configure three illustrative presets with a two-model residency allowance. Request the first two, then the third, then the first again. This tests policy; it does not promise that those models fit a particular GPU. At every step, preserve selected process, loading events, readiness, and response latency. If returning to the first model requires loading, label it a cold path. Comparing that response directly with an already-loaded model as if both timings measured generation alone would misattribute startup work.

Test configuration changes

A changed preset can require reloading rather than merely refreshing a visible list. Preserve old and new configuration, child-process termination, and replacement startup events. Loading or unloading is not equivalent to readiness. Correlate request identity with model state over time. Repeat with a known concurrent arrival schedule where relevant: manual sequential calls do not cover competition affecting unload eligibility and actual client waiting. Keep retries visible so that they cannot disguise a failed initial transition.

Accept the router policy

Keep presets, build, memory map, loaded-model limit, and request sequence. Identify hot and cold paths for each model ID. Do not promise zero cold-start latency or eviction of a busy model unless the tested release supports that behavior. Repeat the scenario when adding a checkpoint with different resource requirements. The result should distinguish configurational availability, residency, and current readiness. This avoids equating an API list with GPU contents and makes latency discussions account for loading policy rather than only token generation from an already running model.

vLLM served-model-name: Fix a Missing API Model; Planning a GPU pool: when is the next server actually needed?.