In this article
When SGLang accepts requests slowly while the GPU is idle, inspect tokenizer and detokenizer workers before changing GPU memory fraction.
When SGLang accepts requests slowly while the GPU is idle, inspect tokenizer and detokenizer workers before changing GPU memory fraction. Input preparation and text assembly can limit service independently of model execution. Diagnosis needs CPU-stage timing and comparable requests. Aggregate processor utilization cannot identify the bottleneck or establish that adding workers will improve the user-facing result.
Locate CPU-stage boundaries
The tokenizer converts text to IDs, while detokenization converts generated output back to text. SGLang server arguments expose separate worker controls and tokenizer-backend settings. Check their semantics in the installed release rather than assuming process count directly multiplies throughput. Available CPU cores do not rule out request transport, serialization, or scheduler waiting as the actual restriction. Localize elapsed time first, then size the worker pool for the stage shown to be limiting progress.
Profile the relevant process
The benchmarking and profiling guide distinguishes CPU/GPU observations and profiling controls. Verify that the trace covers the server rather than only the client’s sending function. Preserve timestamps for arrival, input preparation, execution, and output delivery. For streaming, inspect both initial-fragment delay and subsequent detokenization cadence. A short successful request may miss a problem caused by long inputs or by many small output fragments being handled concurrently. Record the process identity with each trace.
Use contrasting diagnostic inputs
Create one set with long text inputs and short expected answers, and another with short inputs and longer outputs. These are diagnostic profiles, not predictions of server behavior. If the first group stalls before GPU execution, investigate input tokenization. If already-generated fragments in the second group arrive late, inspect reverse conversion and client consumption. Do not change worker counts on both sides simultaneously: an improved result would no longer identify the mechanism responsible for it.
Change settings sequentially
Save a baseline profile, then adjust one worker control while preserving model revision, batch limits, and the arrival schedule. Record CPU contention and memory because additional processes can duplicate state and compete for resources. If GPU utilization rises, check whether user-visible latency improves too. A busier GPU alone is not the objective. Where latency regresses, compare per-process CPU load and waiting stages before moving back to model-kernel configuration or increasing unrelated GPU limits.
Produce bottleneck evidence
Keep traces from before the change, after it, and a repeat of the original configuration. This helps separate tuning effects from warmup or changing inputs. Record tokenizer backend, worker counts, host CPU allocation, and container limits. A single utilization chart cannot prove a CPU bottleneck. Evidence should show that the identified stage constrained intake or delivery and that changing its capacity changed the observed behavior. If tracing instead points to the client or network, preserve that conclusion. More RAM or VRAM does not replace stage-level diagnosis or guarantee that the relevant process can use additional resources.
Related reading
A hybrid CPU/GPU architecture: splitting an AI service by task; SGLang memory fraction: diagnose GPU memory pressure.