In this article
When kubectl drain receives an eviction rejection, inspect the PodDisruptionBudget and healthy replica count. A PDB constrains certain voluntary disruptions through the Eviction API. It neither prevents every failure nor blocks every way of deleting a Pod.
When kubectl drain receives an eviction rejection, inspect the PodDisruptionBudget and healthy replica count. A PDB constrains certain voluntary disruptions through the Eviction API. It neither prevents every failure nor blocks every way of deleting a Pod.
Identify the blocked operation
Save the drain command, API response, budget selector and selected Pod states. Confirm that the selector identifies the intended workload rather than neighboring replicas. Inspect desired replicas, Ready state and allowed disruptions. One replica may already be unavailable following another event. Rejecting an additional eviction is then expected, even when the operator considers maintenance on one more node a small action.
Separate deletion mechanisms
kubectl drain uses eviction and respects PDBs. Direct Pod or Deployment deletion and controller-driven rolling updates follow different rules; a PDB is not a universal deletion lock. Involuntary disruptions also consume available capacity, but the budget cannot prevent network isolation or loss of a node. Do not bypass rejection with direct deletion before understanding the budget and which surviving replicas continue serving work.
Work through three replicas
An illustrative Deployment has three replicas and minAvailable set to two. With every replica healthy, one can be evicted. If one is already not Ready, another eviction may be rejected. These numbers explain budget behavior rather than guarantee continuity for a particular inference service. Investigate why the unhealthy replica cannot recover: readiness failure, resource shortage and an application dependency require different corrections before maintenance proceeds.
| State | Operator response |
|---|---|
| All three Ready | Check permission for one eviction |
| One not Ready | Restore the missing serving capacity |
| Incorrect selector | Correct the budget’s scope |
| Drain waits for a DaemonSet | Investigate the separate drain rule |
Account for unhealthy Pods
unhealthyPodEvictionPolicy affects eviction of unhealthy instances. Documentation recommends considering AlwaysAllow to support maintenance around misbehaving Pods, but the choice must fit the application and cluster version. Do not reduce minAvailable merely to finish a command without assessing remaining capacity. DaemonSets are handled separately: —ignore-daemonsets does not mean their processes have stopped or that the node is otherwise empty. Include that distinction in the maintenance plan.
Finish with verification
After an allowed drain, verify placement and readiness of surviving replicas. Perform the intended maintenance and uncordon the node if it is returning. Preserve the selector, initial Ready state, eviction response and final state as an artifact. Successful command completion establishes the requested set of permitted evictions; it does not replace an application check. Reuse the same evidence format for the next maintenance window so expected budget blocking remains distinguishable from a new scheduling or application problem.
Before parallel maintenance, assess combined eviction demand. Separate drain commands are parts of one capacity change, rather than independent safe actions merely because they run in different terminals.
Related operational reading: gpu exit plan; gpu maintenance change register.