In this article
A full /dev/shm points to a particular shared-memory mount, not automatically to exhausted VRAM. Inspect IPC space and the overall system memory allowance separately. Enlarging tmpfs can move the failure to the memory cgroup if the combined budget is ignored.
A full /dev/shm points to a particular shared-memory mount, not automatically to exhausted VRAM. Inspect IPC space and the overall system memory allowance separately. Enlarging tmpfs can move the failure to the memory cgroup if the combined budget is ignored.
Find the IPC namespace in use
Save the effective —ipc and —shm-size settings of the running container. Inspect its /dev/shm mount from the relevant namespace, including capacity and ownership of objects. IPC modes change the sharing boundary, so comparing only with the host mount can be misleading. Establish whether the application passes shared-memory objects between child processes and which objects survive a worker failure. Do not remove another active process’s objects during investigation.
Check two memory budgets
Docker supports configuring /dev/shm size when the container is created. Kubernetes workloads often use a memory-backed emptyDir for the required mount. Its pages count toward memory use, so a volume sizeLimit does not replace the application’s memory limit. Compare temporary data, process memory and effective cgroup boundaries. Available VRAM cannot substitute for system pages used by shared memory. Observe occupancy while data is being transferred rather than only after the failed process has exited.
Build a reuse scenario
Consider two workers exchanging chunks through shared memory. After unsuccessful starts, objects remain, free capacity falls and another attempt fails. In a separate test environment, compare a successful cycle with one where a child process is terminated unexpectedly. Object closing and removal order matters alongside mount capacity. This example tests cleanup behavior; it does not assert that a particular library has a known leak.
| Evidence artifact | Why to retain it |
|---|---|
| Mount options | Reconstruct the effective IPC configuration |
| Peak occupancy | Calculate temporary data capacity requirements |
| Cgroup memory | Observe the combined system RAM budget |
| Objects remaining after exit | Verify cleanup ownership and completion |
Correct lifecycle and capacity
If stale objects are identified, fix ownership and cleanup before allocating more space. Choose mount capacity from the observed peak with appropriate headroom, retaining a bound on total memory use. Changing —shm-size requires updating and recreating the relevant container. An interactive command does not alter its declared launch configuration. Sharing the host IPC namespace widens process interaction and should be a deliberate design choice rather than a universal capacity workaround.
Test repeatability
Run several complete startup, processing and shutdown cycles using one test dataset. After each cycle, check recovered capacity and leftover objects. Success includes stable residual occupancy and an acceptable peak in memory.current. Preserve the mount definition, worker count, chunk size and verified cleanup behavior. If increasing shared-memory capacity introduces OOMKilled or memory pressure, investigate that newly exposed boundary separately. The earlier IPC failure may have disappeared while the combined system memory allowance remains too small for the workload.
Related operational reading: ram for data preprocessing; gpu containers vs vms.