GPU power caps: an energy and latency experiment

Evaluate GPU power caps as an energy-and-latency experiment using approved, supported operating points. A lower cap does not guarantee less energy per completed task because execution may take longer. Fix workload, measurement boundary, and original settings before comparing alternatives. Do not change limits blindly on shared production nodes or infer savings from one power reading without accounting for completed useful work.

Establish the supported range

The nvidia-smi guide documents power limits and management constraints. Preserve the original mode and the particular device’s permitted range. Reading support does not establish permission or ability to change settings everywhere. A controlled test plan does not need one supposedly universal command. If platform policy prohibits changes, use approved available profiles or state that the experiment cannot be performed. A power cap and direct clock control are related but different decisions; do not substitute one without recording the changed mechanism.

Fix task and work volume

The CUDA Best Practices Guide supports deliberate performance measurement. Preserve prompts, model revision, output policy, and concurrency. Compare equivalent tasks and realized completed work. Shorter answers can consume less energy without indicating better efficiency. Define acceptable TTFT and token-delivery intervals separately so an energy-based choice does not violate application requirements. The objective is useful work within its service contract, not the smallest observed watt value regardless of how slowly or incompletely the request finishes.

Compare operating points

Label tested supported settings A, B, and C. Create columns for cap, elapsed time, completed work, energy, and latency tails. This defines measurement without invented savings percentages. If B draws less power but finishes later, calculate energy for the same accepted workload independently. If C violates a latency threshold, it can be unsuitable even with lower energy. Retain rejected points because they explain the actual tradeoff and selection boundary. Do not report only the most attractive component of each alternative.

Control experimental conditions

Preserve cooling conditions and observable clocks without promising identical boost behavior everywhere. Separate warmup from steady state and repeat the original operating point after the series. Label environmental drift as uncertainty. Use the same sensor boundary throughout: board and AC energy are not interchangeable. Local GPU telemetry alone cannot establish a reduced electricity bill without considering other components, operating duration, and whole-system boundaries. Keep raw observations so calculations can be repeated rather than relying on screenshots of smoothed charts.

Accept an operating point

Keep permission policy, original limit, tested settings, telemetry, and functional fixtures. Document restoration and verify the original configuration afterward. The result should describe a measured energy-latency frontier for one workload, not recommend lower limits universally. Repeat after GPU, workload, or cooling changes. A useful conclusion identifies a setting meeting requirements and its tradeoff cost. It does not promise linear savings, preserved throughput, or equal effects across architectures, and one lower instantaneous reading cannot establish improved end-to-end inference efficiency.

Energy per LLM request: measuring without false precision; Throughput and latency: choosing a GPU operating point.