In this article
When new connections fail while established ones still work, inspect node conntrack. Compare nf_conntrack_count and nf_conntrack_max with failure events. Table exhaustion is one hypothesis among DNS, network policy, socket limits and application failures that can produce similar symptoms.
When new connections fail while established ones still work, inspect node conntrack. Compare nf_conntrack_count and nf_conntrack_max with failure events. Table exhaustion is one hypothesis among DNS, network policy, socket limits and application failures that can produce similar symptoms.
Identify the node and path
Record source Pod, destination, protocol and failure time. Establish where NAT and connection tracking occur in the actual implementation. Kubernetes Services may use kube-proxy or another dataplane, so guidance for one mode is not universal. Compare a test connection through Pod IP and Service IP. Use the comparison to localize the problem without bypassing production access policy for convenience.
Measure table consumption
Kernel nf_conntrack_count reports allocated flow entries, while nf_conntrack_max sets the corresponding boundary. Capture values during failure and afterward. A high count alone does not prove drops; the limit, trend and corroborating evidence matter. Preserve kernel records and available subsystem counters without exporting unnecessary sensitive addresses. A shared node’s entries belong to several workloads, not only the inference worker that first reported a timeout.
Examine short-connection churn
Suppose several test processes create many short connections to an external dependency. Existing flows continue while new attempts become unreliable as table consumption rises. After the burst stops, entries follow their configured lifecycle. This is a testable hypothesis rather than a promise of a fixed release time. Timers depend on protocol, state and kernel configuration and must be checked in the effective environment.
| Observation | Next question |
|---|---|
| Count approaches max | Is there evidence of rejected flows? |
| New flows grow rapidly | Which workload creates them? |
| Only Service routing fails | How is the dataplane behaving? |
| Direct routing also fails | Which shared segment is affected? |
Address the source of churn
Inspect connection reuse and bound background activity without changing application request contracts. Enlarging the table requires assessing memory and shared-node load. Shortening timeouts can disrupt legitimate long-lived connections and is not a universal repair. Change one setting on a test environment and preserve the original. If count is far below the limit and drop evidence is absent, stop tuning conntrack solely because symptoms look similar.
Verify with repeated load
Repeat the same connection profile while recording new-connection success, table consumption and existing-flow health. The artifact should identify node, dataplane mode, timed measurements and the source of churn. Recovery means the confirmed boundary no longer causes failure without damaging required long connections. Reassess scope after a CNI change, since accounting location and available metrics may differ. A networking effect should not be attributed to GPU performance merely because an inference service exposed it.
Related operational reading: network and uptime; inference capacity planning.