In this article
After upgrading vLLM, inspect the effective CUDA Graph mode rather than only the requested setting.
After upgrading vLLM, inspect the effective CUDA Graph mode rather than only the requested setting. Attention-backend capabilities and batch shapes can alter the available capture path. A latency change should not immediately be blamed on the GPU. Preserve configuration, selected backend, and capture logs, then compare decode and mixed steps on one controlled workload before attributing a performance regression.
Interpret mode names precisely
The CUDA Graphs design document describes NONE, PIECEWISE, FULL, and modes with separate decode coverage. Their capture scope and relationship with compilation differ. FULL does not promise that every dynamic operation works for every model. Evaluate the request against the actual runner and supported operations. Keep the current release’s exact configuration setting because an example from an older deployment does not establish unchanged semantics after an engine upgrade. The name alone is not an execution trace.
Check attention compatibility
The attention-backend matrix relates backend selection to model configuration and supported features. Particular implementations can require another execution path for certain shapes. Backend and graph mode are consequently not arbitrary independent switches. Before benchmarking, record the effective backend, dtype, and relevant model dimensions. If an explicit selection is incompatible, keep the rejection reason instead of hiding it through random flag changes that alter several experimental conditions at once. An understandable incompatibility is more useful evidence than an unexplained successful fallback.
Build an acceptance table
For old and new releases, create rows for a single request decoding, several active answers, and prefill mixed with decode. Record requested mode, observed path, capture events, and functional correctness in separate columns. Add latency numbers only after measurement. If the discrepancy occurs solely in the mixed row, investigate support for that batch shape. Do not conclude that FULL has stopped working globally or that the entire model is unsupported because one execution category takes a different route.
Separate capture from warmup
The first execution can include compilation and graph preparation. Compare repeated runs separately from cold startup. Preserve context settings, output lengths, and request-arrival schedules because a different shape distribution changes the paths actually exercised. Use profiling to corroborate the interpretation of log messages. Low average GPU utilization does not identify the selected graph path and cannot substitute for capture-event evidence from the process being investigated. Likewise, a warmed filesystem cannot establish that graph preparation was reused.
Record the release decision
The manifest should include mode, backend, model revision, and tested shapes. If fallback is acceptable, document its scope and measured cost for that workload. If a strict execution mode is required, incompatibility should fail acceptance rather than silently weaken the requirement. Successful capture alone does not promise acceleration; graph launches can account for different fractions of total latency. Restore the agreed runtime and settings during rollback. The resulting path table lets another operator distinguish an expected implementation change from a configuration mistake without assuming that an unchanged launch-command string guarantees an unchanged computation route.
Related reading
Install vLLM: Check GPU, Driver and Package Compatibility; Updating an LLM on a GPU: testing a new version with a small rollout.