In this article
An unreachable node does not cause every Pod to move immediately. Kubernetes detects the issue, updates conditions and taints, then applies tolerationSeconds and eviction rules. Reconstruct the actual sequence before treating slow recovery as a scheduler defect.
An unreachable node does not cause every Pod to move immediately. Kubernetes detects the issue, updates conditions and taints, then applies tolerationSeconds and eviction rules. Reconstruct the actual sequence before treating slow recovery as a scheduler defect.
Preserve both sides of the state
Record Node Ready, recent condition transitions, taints and Pod identity. An application may continue running while its node cannot reach the control plane. Missing communication does not prove process termination. This matters especially for operations with external side effects. Do not create a duplicate manually merely because a dashboard has lost the heartbeat. Establish the workload’s rules for stopping ownership and repeating work first.
Inspect tolerations
Kubernetes normally adds NoExecute tolerations of 300 seconds for not-ready and unreachable unless explicitly configured otherwise. This represents waiting after detection, not a promised recovery time measured from physical failure. DaemonSets have different tolerations. Controller settings, eviction rate limits and zone health also matter. Match documentation to the installed version: taint eviction architecture has changed, and detection time is additional to the toleration interval.
Examine a network partition
In an example, a node loses contact with the control plane, Ready becomes Unknown and the Pod remains bound until its allowance expires. Even after eviction, another node may lack a suitable GPU. The complete interval includes detection, toleration, deletion, scheduling and replacement preparation. Reducing one value does not shorten every stage or establish that the original process has stopped.
| Stage | Required evidence |
|---|---|
| Detection | Node condition transition |
| Waiting | NoExecute taint and effective toleration |
| Eviction | Controller event and Pod state |
| Replacement | New UID, scheduling and readiness |
Adjust policy with isolation in mind
A short allowance may accelerate replacement while causing unnecessary movement during brief outages. A long allowance preserves node attachment but delays response to genuine loss. Stateful operations need additional ownership and protection against simultaneous execution. Force deletion removes the API object without confirming process termination, so it cannot establish resource release. Describe an acceptable partition scenario rather than choosing only a desired latency number.
Measure the complete timeline
In a controlled environment, separately test brief disconnection and sustained unavailability within permitted impact boundaries. Save conditions, tolerations, eviction events and actual replacement readiness. After the node returns, inspect old processes and resource assignments. The artifact should show where time was spent and which conclusions have evidence. It distinguishes an intentional waiting policy from GPU shortage, application startup failure or inconsistent post-partition state. Keeping these stages separate makes a future recovery-time comparison meaningful instead of attributing every delay to tolerationSeconds.
Related operational reading: network and uptime; gpu owner network outage log.