# Tune SGLang schedule conservativeness for output-length uncertainty

Tune SGLang schedule conservativeness around uncertainty in future answer length, not as a substitute for the GPU memory budget.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/sglang-schedule-conservativeness/
Language: en

Tune SGLang schedule conservativeness around uncertainty in future answer length, not as a substitute for the GPU memory budget. A large `max_new_tokens` limit can accompany a short realized answer, so reservation estimates may leave capacity unused. Conversely, aggressive scheduling can increase retractions. Accept a setting from queue and execution observations on a preserved input-and-output length profile.

## Understand scheduler reservation

The engine must consider both current tokens and possible continuation of accepted requests. The [SGLang tuning guide](https://docs.sglang.io/docs/advanced_features/hyperparameter_tuning) explains scheduling conservativeness through token usage, waiting work, and retractions. This differs from the physical memory fraction assigned to the runtime. One control changes expectations about future work; the other establishes a resource boundary. They are not interchangeable ways of achieving the same outcome, even when both appear to change the number of requests served concurrently.

## Preserve output-length distributions

Record the requested output ceiling and actual generated-token count for each request. If the application routinely sets a large ceiling but the model usually reaches EOS early, retain that characteristic. Do not shorten the ceiling merely to improve a benchmark if it changes the real requirement. Separate short and long completions and preserve their proportions. During repeats, verify that varying answers have not changed the workload enough to invalidate comparison between settings. The same input list can still produce different realized work.

## Use a diagnostic branch

Suppose the queue grows, token usage remains modest, and most answers finish well below their permitted budget. That motivates investigating reservation; it does not prove the diagnosis. Frequent retractions under high occupancy suggest examining the other side of the tradeoff. Populate a table with queued work, token usage, completions, retractions, and latency. Align observations to one interval and configuration. Combining queue data from one run with retraction data from another creates a misleading apparent relationship.

## Check causality with profiling

Use the [official profiling guide](https://docs.sglang.io/docs/developer_guide/benchmark_and_profiling) to complement counters with an execution timeline. Change conservativeness independently of memory fraction, active-request limits, and chunked prefill. Retain an identical arrival schedule for each series. If throughput rises, inspect latency tails and repeated work as well. An aggressive configuration can improve one aggregate while making service less stable. Also check whether answers became shorter: doing less actual work is not evidence that the scheduler became more efficient.

## Define an operational boundary

Choose the setting that meets queue and latency requirements for the evaluated profile. Keep its value, engine version, length distributions, and rejected alternatives. A result is not universal across models with different memory behavior or termination patterns. Repeat the evaluation after application output-policy changes. The artifact should connect reservation assumptions to scheduling behavior and distinguish both from memory allocation. This lets another operator understand why a cautious or aggressive setting was selected instead of repeatedly raising memory fraction when unused capacity actually comes from estimated future answer length rather than unavailable VRAM.

## Related reading

[SGLang memory fraction: diagnose GPU memory pressure](/journal/sglang-memory-fraction/); [Planning a GPU pool: when is the next server actually needed?](/journal/inference-capacity-planning/).