In this article
OOMKilled and Evicted call for different responses. The former generally describes a container termination associated with exhausted memory; the latter describes a kubelet decision to evict a Pod under node pressure. An exit code alone is insufficient: reconstruct container state, the Pod reason and node events.
OOMKilled and Evicted call for different responses. The former generally describes a container termination associated with exhausted memory; the latter describes a kubelet decision to evict a Pod under node pressure. An exit code alone is insufficient: reconstruct container state, the Pod reason and node events.
Preserve state before recreation
Capture the Pod YAML, describe output and timestamped events. For each affected container, inspect state, lastState, reason, exitCode and restartCount. At Pod level, preserve phase and reason. Deleting the object can remove valuable local context, so repeated restarts should not be the first diagnostic step. Record nodeName, Pod UID and container ID too. A Deployment name can cover successive instances with different termination causes.
Identify the decision boundary
When a cgroup memory limit is exceeded, the kernel may kill a process. If the main container process exits, the kubelet applies the configured restartPolicy. Node-pressure eviction instead uses node conditions and terminates the Pod, after which its controller may create a replacement. A node can also reach OOM before the kubelet successfully reclaims resources. The phrase “ran out of memory” therefore does not identify which boundary needs adjustment or where the original event should be found.
Separate two illustrative cases
In the first example, a container has a 4 GiB limit, a worker expands its working set and receives OOMKilled while other Pods continue running. In the second, the container remains within its own limit, but node memory.available falls and the kubelet reports eviction. The 4 GiB value illustrates a constraint rather than recommending memory capacity for a model. Each case needs independent corroboration from events and memory accounting.
| Evidence | Investigation scope |
|---|---|
| OOMKilled in lastState | Container memory and kernel records |
| Evicted as the Pod reason | Pressure signals and kubelet decisions |
| A new UID on another node | Controller-created replacement |
| Increasing restartCount | Container restart within the same Pod |
Correct the relevant boundary
For a cgroup OOM, assess peak working set, cache and memory-backed volumes, then check the effective limit. For eviction, investigate reservations, requests, neighboring workloads and node pressure signals. Simply raising a limit can intensify the shared shortage. Changing QoS for its label alone does not replace resource planning. If another pressure category triggered eviction, do not automatically apply a RAM diagnosis to local storage or PID pressure.
Verify recovery behavior
Reproduce the scenario in an isolated environment and record which mechanism creates the replacement: container restartPolicy or a workload controller. Ensure observability preserves the termination reason before the Pod disappears. A successful correction includes normal startup and absence of the same event under comparable load. The incident artifact should connect the Pod UID, memory boundary, node condition and chosen action. This lets the next operator distinguish a recurrence of the original problem from a different failure hidden behind a similar restart symptom.
Related operational reading: kubernetes gpu scheduling; memory optimized hosting.