Manage vLLM torch.compile caches across releases

A vLLM compilation cache can shorten a compatible repeat startup, but copying its directory does not establish compatibility with a new release. Preserve the cache’s dependencies and test cold and repeated starts separately. Python code, runtime configuration, and library changes may require recompilation. This is startup-artifact management, not a guarantee of identical readiness times across servers or GPUs.

Identify cache dependencies

The torch.compile integration document explains cache dependencies on configuration and code. A directory produced by a successful run therefore belongs to a particular set of conditions. Preserve the engine version, PyTorch configuration, model revision, and compilation settings. Matching model names alone is insufficient: infrastructure code can change the computational graph without changing a checkpoint. The manifest should state which environment components were checked and how reuse was verified, rather than describing a directory as universally reusable.

Separate warming layers

Weight loading, graph compilation, and CUDA Graph capture are different stages. The CUDA Graphs design describes capture modes and their relationship with execution partitions. A faster second start may result from the operating system’s file cache even if compilation artifacts were not reused. Retain stage timestamps and the corresponding log messages. A separate Triton compilation cache should also not be confused with Triton Inference Server: it stores kernel-compilation artifacts rather than a network model repository.

Run a startup matrix

Create four records: the original release without warming, its repeat run, the new release with transferred artifacts, and a repeat of the new release. This defines an experiment rather than supplying predicted durations. Record cache locations, actual compilation events, and time to an acceptance request. If the new release rebuilds the cache, inspect documented invalidation conditions. Safely declining an incompatible artifact does not establish a broken deployment; it may be the expected response to changed code or configuration.

Package artifacts deliberately

Include an environment manifest and a list of files belonging to the selected cache. Record ownership and path availability inside the serving container. Before timing, verify that the process reads the intended directory instead of silently creating another cache elsewhere. Avoid mixing outputs from unidentified parallel warmups. Without run identifiers, an operator cannot establish which configuration produced an artifact or whether a later startup is entitled to treat it as compatible. Keep source and destination locations visible in the experiment record.

Accept behavior and retain rollback

Run a functional request after startup, followed by representative batch shapes. An absence of compilation messages cannot replace response validation. Preserve the previous configuration and a clean-warmup procedure for rollback when old artifacts are unavailable. Distinguish demonstrated cache reuse from an observed fast startup in the report. Repeat the matrix after runtime upgrades instead of turning one successful transfer into a universal rule. The result should explain why ready-made code was reused or compilation ran again, and attribute measured time to those stages without quietly combining it with file-cache effects.

Related reading

Reproducible inference stacks: preserving a GPU pilot’s result; GPU autoscaling: a free card is not a ready replica.