# GPU Deployment rollout stuck: maxSurge without spare capacity

Diagnose gPU Deployment rollout stuck: maxSurge without spare capacity. Evidence to collect, a worked operational example and recovery checks.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/kubernetes-gpu-rollout-surge-deadlock/
Language: en

A GPU Deployment rollout can remain Pending when the new replica needs spare capacity and every suitable device is occupied by old Pods. Inspect maxSurge and maxUnavailable: permission to create a Pod does not provide the resources needed to run it.

## Confirm the waiting stage

Preserve Deployment conditions, ReplicaSets and Pending Pod events. Separate FailedScheduling from image pulling and failed readiness. Compare old and new resource requests, node constraints and available capacity. A GPU elsewhere may not satisfy affinity or host-memory requirements. This investigation concerns replacement at full capacity, rather than initial device-plugin installation or basic GPU resource declaration.


[Official reference 1](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/).

## Calculate permitted overlap

maxSurge bounds additional Pods; maxUnavailable bounds temporarily unavailable desired replicas. Percentages use different rounding rules, so calculate actual integers, especially for a single replica. Terminating Pods can continue consuming resources. Object count alone does not prove that a device has been released. A progress deadline reports stalled progress, but does not manufacture capacity or automatically repair the rollout strategy.


[Official reference 2](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/).

## Examine one occupied accelerator

In an illustrative cluster, one replica occupies the only suitable GPU. maxSurge 1 and maxUnavailable 0 create another replica that cannot run while preserving the original. This is an expected capacity constraint. Without another suitable GPU, the transition must permit temporary unavailability or use another explicitly designed approach. Changing these values does not promise an interruption-free rollout or immediate device release when a Pod is deleted.

| Option | Planning consequence |
| --- | --- |
| A spare suitable GPU exists | Overlapping replicas can be tested |
| No spare capacity exists | Plan a permitted sequential replacement |
| The old Pod is terminating | Wait for actual resource release |
| The new Pod is Pending | Read its specific scheduling reason |

## Choose the strategy deliberately

Define acceptable interruption, preparation time and rollback behavior. If overlap is required, provide suitable capacity beforehand. If sequential replacement is acceptable, calculate rollout parameters and test them on a representative workload. A PDB is not a universal constraint on Deployment rolling updates; these mechanisms have different scopes. Deleting arbitrary neighboring Pods to free a device bypasses their ownership and operating requirements.

## Verify rollout and rollback

Retain a timeline covering ReplicaSet creation, Pending state, old-process exit, device assignment and new-replica readiness. Test return to the previous digest using the same capacity policy. Success means a predictable transition within the selected interruption allowance. The artifact should contain integer surge/unavailable values and observed waiting reasons. It makes strategy errors distinguishable from image failures and prevents a successful controller status from obscuring the capacity required to repeat the transition. Reassess the calculation whenever the new template changes CPU, RAM or device requirements, even if replica count stays unchanged.

Record actual device release separately from API deletion. Those timestamps distinguish controller transition from waiting for the old execution environment to finish.

Related operational reading: [kubernetes gpu scheduling](/journal/kubernetes-gpu-scheduling/); [model update canary](/journal/model-update-canary/).