In this article
Too many open files requires checking the limit of the process that actually failed. The ulimit of a new shell, global file-max and an existing worker's allowance belong to different layers. Raising the wrong boundary will not correct a leak or a faulty launch configuration.
Too many open files requires checking the limit of the process that actually failed. The ulimit of a new shell, global file-max and an existing worker’s allowance belong to different layers. Raising the wrong boundary will not correct a leak or a faulty launch configuration.
Preserve the PID and failure time
Identify the process through its container ID and active instance. With appropriate permission, inspect /proc/PID/limits and count entries in /proc/PID/fd. Avoid publishing descriptor contents unnecessarily: they may expose sensitive paths or sockets. Record both soft and hard open-file limits. An interactive shell reports its own allowance, which can differ from the inherited limits of the main process already running in the container.
Separate system and process ceilings
Kernel file-max bounds system-wide file handles, while file-nr exposes related accounting and nr_open constrains the process limit ceiling. RLIMIT_NOFILE applies separately. Compare the application error with kernel records and host-wide consumption. System exhaustion can affect several containers. A process limit can prevent one worker from opening connections while global capacity remains available. Free GPU memory cannot establish which of these cases occurred.
Build a sequence of observations
Suppose a worker starts with a soft allowance of 1024 descriptors. After repeated connections, consumption approaches that boundary and does not fall after work completes. Investigate unclosed files or sockets. If usage returns to baseline but legitimate peak demand exceeds 1024, investigate the capacity of the intended workload. The number is illustrative rather than a universal inference-server recommendation.
| Measurement | Question answered |
|---|---|
| Before load | What is the process baseline? |
| At peak load | How many handles are needed concurrently? |
| After completion | Are handles being released? |
| After repetition | Is consumption growing cumulatively? |
Correct the source of consumption
Group descriptors by purpose and lifecycle owner: client connections, files, pipes and event objects. Fix cleanup or pool behavior if usage grows without a bound. For justified capacity, declare nofile in the Docker launch configuration and verify the recreated container. Uncontrolled increases for one worker can move exhaustion to the host. Changing the current shell is not evidence that the service configuration has been updated.
Verify remaining headroom
Repeat the complete sequence with the same connections and files. Recovery requires successful opens, stable residual consumption and sufficient separation between the observed peak and soft limit. Preserve the process identity, image digest, launch setting and consumption graph. The PID will change at the next startup, so the validated configuration is the durable identifier. If failure persists with ample process headroom, return to system-wide handles and surrounding components. Continuing to raise the same number would obscure the distinction that the investigation was intended to establish.
Check whether sockets disappear after client sessions close. This distinguishes a legitimate peak of concurrent connections from resources retained beyond their intended lifecycle.
Related operational reading: server preparation; gpu containers vs vms.