# Read-only container will not start: where to allow temporary and cache writes

Diagnose read-only container will not start: where to allow temporary and cache writes. Evidence to collect, a worked operational example and recovery checks.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/read-only-container-writable-caches/
Language: en

A read-only root filesystem can expose hidden writes to /tmp, caches, PID files or home directories. Identify the exact path and owner. Permit writing through bounded dedicated mounts while retaining an immutable root, rather than removing the restriction entirely.

## Preserve the first write failure

Record path, operation, UID/GID and startup stage. Separate EROFS from ordinary permission denied: read-only mounts and incorrect ownership require different corrections. Inspect effective securityContext and mounts after admission, not just the Dockerfile. Libraries may write during import or cache preparation before the main loop. Do not attribute failure to CUDA before locating the system operation and filesystem scope.


[Official reference 1](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/).

## Classify the data

Temporary files may use a bounded writable tmpfs or emptyDir with an appropriate policy. Durable artifacts need a persistent lifecycle. Secrets and trusted inputs should remain read-only when the process must not change them. Docker tmpfs has temporary semantics and limitations, including possible interaction with host swap. Memory-backed mounts consume system RAM, and writable capacity does not replace the process memory budget.


[Official reference 2](https://docs.docker.com/engine/storage/tmpfs/).

## Examine a minimal startup

Suppose a read-only container tries to create /run/worker.pid and /tmp/session. Add separate writable mounts with suitable UID/GID and bounded sizes, then repeat startup without changing other settings. If a library also writes to a home-directory cache, locate its documented path configuration or provide another scoped mount. This maps writes rather than recommending caches or tuning parameters for a specific engine.

| Data path | Required question |
| --- | --- |
| PID/runtime files | Is writing needed only for process lifetime? |
| Temporary data | What are peak capacity and cleanup rules? |
| Cache | Can contents be recreated safely? |
| Model/artifact input | Should the process modify these files? |


[Official reference 3](https://docs.docker.com/reference/cli/docker/container/run/).

## Grant minimal permissions

Permit only necessary directories and account for ownership after mounting. A writable mount with the wrong UID still fails; broad chmod changes a different boundary. Consider tmpfs nosuid/nodev/noexec options according to purpose. Noexec can break legitimate execution from temporary files and must be tested. Declare mode and size for every mount so a manually created directory does not become a hidden startup requirement.

## Verify cleanup and recreation

Process test data, shut down and create a new container without previous temporary state. Confirm successful startup, no unexpected root writes and acceptable memory peaks. Preserve paths, permissions, mount lifecycle and repeated results. Recovery means known writable boundaries, not merely suppressing an error. Recheck after image upgrades, since a library can introduce another cache path without changing the external launch command. The resulting artifact ties isolation to actual application behavior and makes future failures easier to localize without progressively broadening the writable filesystem.

Related operational reading: [gpu containers vs vms](/journal/gpu-containers-vs-vms/); [hardware security](/journal/hardware-security/).