CUDA Graphs vs torch.compile: replay and compilation can coexist

CUDA Graphs and torch.compile are often presented as alternative inference accelerators. Compilation can transform how a region executes; CUDA Graph replay can reduce the overhead of launching a supported sequence of GPU work. They can be used together. The right question is which overhead exists in the actual service and which mechanism addresses it.

Separate transformation from replay

The torch.compile reference describes backend-driven optimization and modes with different tradeoffs. Compilation is not simply remembering a previous answer, and it does not remove the need to execute the model for a new request.

The CUDAGraph Trees documentation explains replay and its constraints. A captured sequence has requirements involving operations, dependencies and memory addresses. General arbitrary control flow is not automatically replayable. Read the support of the selected framework version rather than assuming any Python function becomes one reusable GPU graph.

Find the limiting overhead

Use profiling to distinguish GPU computation from CPU launch gaps, graph breaks and first-use compilation. The official torch.compile profiling guide provides the framework context for reading these events.

A workload dominated by large matrix computation may have a different opportunity from a small-batch workload spending substantial time between kernel launches. If request queueing dominates observed latency, optimizing launch overhead alone may barely change the user’s experience. Name the measured bottleneck before changing modes.

Account for shape and control diversity

An LLM server handles changing prompt lengths, batch sizes, decoding lengths and sometimes different adapters. Record which shapes the engine compiles or captures and which use another path. A single repeated shape is useful for diagnosis but not a full production trace.

Test a representative sequence of shapes in realistic order. Look for repeated compilation, capture churn and memory growth. Changes that improve one warm case can make the overall worker less predictable if many requests continually introduce new specializations.

Compare startup and steady state separately

Measure worker readiness, first request, warmup and sustained serving. Keep the model, precision, attention backend and arrival pattern constant. Confirm that the intended optimization actually ran before attributing a difference to it.

Count useful completions and latency tails along with memory usage. For a long-lived worker, startup cost may be amortized over many requests. For frequently replaced workers, that same cost can consume a material portion of available capacity. Report both scenarios instead of hiding initialization outside every chart.

Select a documented operating mode

Choose a supported engine configuration rather than layering unrelated compiler flags until the server starts. Preserve a baseline path for correctness comparison and unsupported shapes. Task output checks still matter when numerical execution changes.

The conclusion should identify the tested model, framework release, shapes and worker lifetime. It should explain whether gains came from compiled kernels, replay, reduced Python work or another interaction. Avoid multiplying separate headline speedups: optimizations that target overlapping overheads do not automatically compound, and their memory costs can alter the concurrency a worker can sustain.

Related reading

Manage vLLM torch.compile caches across releases, Detect vLLM CUDA Graph mode fallbacks after upgrades.