In this article
When a production image lacks a shell and network utilities, use an ephemeral debug container with appropriate permissions. It supplies a diagnostic environment without necessarily rebuilding the application image. This still changes Pod state and calls for a deliberate choice of image and capabilities.
When a production image lacks a shell and network utilities, use an ephemeral debug container with appropriate permissions. It supplies a diagnostic environment without necessarily rebuilding the application image. This still changes Pod state and calls for a deliberate choice of image and capabilities.
Define the diagnostic question
First preserve events, container state and available logs. Network investigation needs the relevant network context; process inspection needs the appropriate process namespace. Do not grant every privilege merely to test DNS or a port. Specify permitted data access and select a trusted tool image at a verified version. A minimal production image remains useful even when an investigation temporarily needs additional tools.
Check runtime support
kubectl debug can add an ephemeral container to a running Pod. —target selects another container’s process namespace, but runtime support is required. Without it, the debugger may not see the intended processes. Ephemeral containers have lifecycle and resource restrictions; they are not ordinary permanent sidecars. Check Kubernetes version, debug profiles and user authorization before launching, keeping an access denial distinct from the original application failure.
Run a bounded example
In a test Pod, the main image contains no sh and kubectl exec fails because the executable is absent. A separate debug container can inspect resolver addressing and connectivity to the intended Service. A suitable target enables process investigation. Record whether the expected PID is visible and where the network command ran. A successful debug request does not prove that the application uses identical credentials, CA trust or proxy settings.
| Investigation scope | Evidence to preserve |
|---|---|
| Networking | Namespace, destination and port |
| Processes | Target and confirmed PID |
| Data access | Permitted mounts and UID |
| Tools | Image digest and debug profile |
Choose how to reproduce safely
If reproduction requires changing launch commands or resources, a Pod copy may be appropriate. That copy is another workload and can add load or side effects. Do not start it with production secrets and active processing without checking its purpose. On a running Pod, prefer a minimal diagnostic toolset and retain results outside the interactive session. Closing a terminal does not automatically erase the added container from Pod state.
Preserve findings and end access
Record the original question, commands without secrets, namespace scope and outcome. Separate confirmed observations from hypotheses the tool could not test. End the session and clean up a temporary copy if one was created. The artifact should permit repetition with the same image digest. This recovers evidence without permanently expanding the production image or leaving an undocumented administrative channel. Review the captured scope when another engineer interprets results, because a command that worked in the debugger may still differ from the application’s actual execution environment.
Related operational reading: gpu containers vs vms; vllm localhost container port.