TECHNOLOGY

High latency at moderate utilization: diagnosing Linux PSI

In this article
THE QUICK TAKE

PSI reports time spent waiting for resources, which can be hidden behind moderate average utilization. Compare pressure at host and process-cgroup scope when investigating latency. These measurements help locate CPU, memory or I/O stalls, but they do not directly diagnose the GPU.

PSI reports time spent waiting for resources, which can be hidden behind moderate average utilization. Compare pressure at host and process-cgroup scope when investigating latency. These measurements help locate CPU, memory or I/O stalls, but they do not directly diagnose the GPU.

Choose the observation scope

On supported Linux systems, host measurements are exposed through /proc/pressure/cpu, memory and io. Cgroup v2 provides corresponding pressure files for individual groups. Host figures combine workloads across the machine, whereas local figures describe the selected hierarchy. Preserve both scopes and the worker path. A lightly loaded container can still wait because neighboring workloads contend for a shared resource. A dashboard value without its scope cannot explain that situation unambiguously.

Official reference 1.

Distinguish some, full and time windows

Some means at least some tasks were stalled on a resource. Full represents simultaneous stalls across all non-idle tasks in the relevant scope. System-level CPU full has special semantics, so do not transfer the interpretation of memory full mechanically to that field. Avg10, avg60 and avg300 cover different windows; total accumulates stall time in microseconds. Verify the fields supported by the installed kernel and compare changes rather than interpreting an absolute total in isolation.

Official reference 2.

Connect waiting with an application stage

In an illustrative case, data preparation slows down, the GPU waits intermittently, and memory some and full increase inside the worker cgroup. memory.events high increases at the same time. Investigating reclaim and memory.high is a reasonable next step. If io pressure rises without a matching memory event, the next investigation differs. These observations form a diagnostic example rather than a measured performance claim for a distributed pool or cloud instance.

Recorded fieldPurpose
PSI scopeSeparate node behavior from workload behavior
avg10/60/300 windowDistinguish bursts from sustained episodes
Change in totalCompare equal observation intervals
Application stageConnect stalls with preparation or serving

Test the hypothesis with one change

After selecting a resource, inspect its actual constraints: CPU quota, memory-controller boundaries or the I/O path. Reduce competing work on a test node and repeat the same inputs. Falling PSI alongside improved latency supports the hypothesis, but does not replace an explanation of the mechanism. Avoid a universal acceptable percentage. Sensitivity depends on the process, window and operation being measured. Comparing values from different scopes needs an explicit explanation.

Preserve a timeline

The final artifact should combine workload start, stalls, cgroup events and operation completion. Include units and collection intervals. After a correction, compare recurring pauses as well as average duration on the same workload. If delays remain while PSI is quiet, investigate network waits, application locks and other mechanisms. Low pressure does not certify the whole service as healthy; it narrows the investigation. Retaining this distinction prevents a resource dashboard from becoming a misleading substitute for evidence about actual worker progress.

Related operational reading: gpu monitoring scorecard; cpu vs gpu hosting.

← All articles