# Identify which scheduler owns batching in Triton and TensorRT-LLM

For Triton and TensorRT-LLM, first identify which component forms batches in the selected deployment path.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/triton-llmapi-batching-boundaries/
Language: en

For Triton and TensorRT-LLM, first identify which component forms batches in the selected deployment path. Generic Triton dynamic batching, current LLM API settings, and older inflight configurations are not interchangeable. A `max_batch_size` field does not describe the entire scheduler. Preserve backend and model configuration before testing actual request combination and waiting behavior instead of copying nearby examples without checking scope.

## Identify the deployment path

The [current LLM guide](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/getting_started/llm.html) uses the LLM API/PyTorch path and places `triton_config` in `model.yaml`. Its published example explicitly describes `max_batch_size: 0` as serving without batching. That describes the example, not a universal prohibition on TensorRT-LLM batching. Establish whether this is the path in use before applying its rules. Matching the model name `tensorrt_llm` does not prove two deployments share a backend, scheduler, or input-tensor schema.

## Separate scheduler layers

The [Triton batcher guide](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/batcher.html) describes dynamic batching for supported stateless models and per-model queue controls. An LLM executor can have separate scheduling logic. Adding a generic block copied from another backend does not establish the intended inflight behavior. Record request acceptance, work formation, and applicable limits at each boundary. Do this before choosing batch-size or queue-delay numbers. Otherwise, a numerically plausible configuration can target a layer that never performs the intended batching operation.

## Map responsibility explicitly

Create rows for the Triton frontend, model configuration, and LLM execution. Record input unit, supported combination behavior, settings, and observation source. This is an inventory template, not a claim that every deployment batches at all three levels. Keep a separate record for a legacy path. Do not transfer `preferred_batch_size` or older scheduler policies automatically: similar names and a shared throughput objective do not establish equal semantics or compatible configuration. Preserve the source supporting each intended control.

## Observe actual execution

Submit a saved concurrent request sequence with known lengths. Correlate execution evidence, queue time, and response latency available from the backend. Client parallelism alone does not prove requests formed a batch. Repeat after changing one supported parameter. If the configuration still serves without batching, lack of benefit cannot show that batching is universally useless. First confirm that the intended mechanism was enabled and actually processed the test requests. Keep unsuccessful configuration acceptance distinct from an unfavorable measured performance result.

## Bound the conclusion

Retain repository layout, runtime version, configuration, and launch commands. Identify settings owned by Triton and those belonging to the LLM API. Measure waiting cost separately from computation benefits; larger batches do not promise lower latency for every request. Rebuild the responsibility map when migrating a legacy deployment instead of mechanically copying its config. The result should explain where work is combined and what evidence supports that account. It should not collapse every path into one universal batch-size formula or assume that a multithreaded client automatically receives continuous batching.

## Related reading

[Continuous batching: why concurrent requests change GPU value](/journal/continuous-batching/); [Planning a GPU pool: when is the next server actually needed?](/journal/inference-capacity-planning/).