# GPU autoscaling: a free card is not a ready replica

Measure GPU autoscaling from allocation through weights, preparation and useful output, including queue behavior and a failed-start retry.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/autoscaling-gpu-cold-start/
Language: en

Growing traffic suggests adding GPUs automatically, but several stages separate the expansion decision from model readiness. Counting machine startup alone can overestimate responsiveness. Owners of idle nodes and partners paying for reserve capacity need the complete path to useful service.

## Break startup into stages

Record node allocation, environment startup, weight retrieval, memory preparation and model validation, each with start, completion and error detection. [Cloud Run GPU support](https://cloud.google.com/run/docs/configuring/services/gpu) is a particular managed capability. [Minimum instances](https://cloud.google.com/run/docs/configuring/min-instances) separately preserve readiness. Neither describes every autoscaling platform's behavior.

An owned server still needs its environment and model prepared even when the physical GPU is already installed.

## Test different initial states

Distinguish a machine not yet allocated, a running host without a loaded model and a model ready for requests. Do not average them into one startup time. Run the same illustrative task from each and record the first useful answer, not merely successful process creation.

Test without a local weight copy separately. Network artifact retrieval can change the outcome; a warmed disk should not remain a hidden condition behind the best measurement.

## Define queue behavior

Requests can arrive while a new replica prepares. Agree waiting limits, rejection and possible movement to another available resource. Do not leave the queue unbounded merely to show an attractive absence of errors.

Test a post-idle burst and gradual growth. Identical total request counts can require different reaction times. Record user cancellations separately: generating an abandoned result can occupy a GPU without completing useful work for that task.

## Compare reserve with repeated startup

A ready replica creates idle costs; cold starts create delays and repeated preparation. Compare calendars for several demand profiles. Reserve is neither universally excessive nor universally necessary: the decision depends on waiting tolerance, resource availability and model.

Show funders which readiness level is financed and how it is verified. An inventory entry does not prove an endpoint can accept traffic immediately. Include another attempt after failed preparation: an allocated machine may fail to retrieve an artifact or pass quality checks. Establish when attempts stop, an owner is notified and the request receives a clear status. This measures useful readiness rather than only successful creation.

## Node readiness criteria

Describe power-on mode, schedule and remote-maintenance capability.

Prepare with [serverless versus ready replicas](/journal/serverless-vs-always-on-inference/), [NVMe for models](/journal/nvme-for-model-serving/) and [GPU-pool capacity planning](/journal/inference-capacity-planning/).