In this article
Graceful shutdown requires several timers and layers to agree. TERM and readiness alone do not guarantee every long response completes.
Restarting an API process supports updates, but active generations can last longer than ordinary HTTP work. Immediate exit leaves partial answers; waiting forever prevents rollout completion. Graceful termination is therefore a bounded protocol: stop admitting work, allow accepted requests a defined interval, classify remaining outcomes and release resources.
Describe the timeline
Kubernetes documents termination flow, including preStop, process signals, grace periods and endpoint state. These stages relate but are not one atomic application operation. A retiring instance should stop receiving ordinary new traffic while handling open connections under its chosen policy.
Draw stop-admission, draining start, final completion deadline and forced-stop points. These are proposed service states rather than mandatory OpenRouter events. Give each an observable signal so tests do not rely on operator assumptions.
Align application timers
The vLLM serve reference documents its shutdown timeout. Verify applicable settings and behavior for the installed version. A parameter name does not establish agreement with orchestration grace periods or load-balancer timeouts.
When the internal duration exceeds the external one, the process can die before expected cleanup. When shorter, the application ends work before the available outer window expires. Keep every timer in one record, naming its owner and the event starting its countdown.
Check accepted requests
Start several short synthetic streams and one bounded longer stream. Initiate normal termination of the test instance. Record acceptance, final event, terminal outcome and client state for each operation. An aggregate HTTP success counter cannot establish individual-response completeness.
Send a new test request after draining begins. Its outcome should follow the declared admission policy. Do not require one universal rejection status across layers: a load balancer may select another instance while an application rejects locally. The report should expose where handling actually occurred.
Account for transport connections
RFC 9113 describes HTTP/2 streams and connection shutdown mechanisms. One connection may carry several operations, so closing it is not one model terminal outcome. Where proxies change protocols, investigate downstream and upstream paths separately.
A long response exceeding the grace window needs explicit classification. It does not become completed merely because TERM was handled correctly. Preserve stopping reason and the final confirmed boundary rather than promoting partial text into a full result.
Repeat the recovery path
After the new version starts, check model readiness and a small control request. This covers the update cycle: removing the old instance, bringing up the new one and exposing the selected interface. It proves neither universal fault absence nor abrupt-failure handling.
Related guides cover model canaries and fallback routing. A termination artifact contains timelines, settings and per-request outcomes, supporting evaluation of one rollout without a universal SLA or production-infrastructure claim. Retain the deliberately long fixture too, because a release tested only with brief requests can conceal the exact deadline conflict the procedure is meant to reveal.