Handle final completion correctly in Triton decoupled streaming

In Triton decoupled mode, identify completion through the protocol signal rather than the last nonempty text fragment. One request can produce multiple responses or no ordinary useful response, and responses from different requests can arrive out of order. A client state machine must handle this contract or risk closing early, losing completion, or waiting for text that is no longer required.

Understand decoupled transactions

The Triton documentation describes independent response counts and explicit completion. This differs from assuming one request produces exactly one response. Both backend and frontend must support the selected mode. Do not generalize ordinary inference-endpoint behavior to every streaming transport. Establish protocol, endpoint, and client version before mapping events onto request lifecycle. Text chunks alone cannot describe that complete contract, particularly when control messages carry no displayable content.

Separate payload from completion

An empty final event can carry control meaning without extending the displayed answer. Preserve events before filtering empty chunks. A client that removes them first may discard the required completion signal. With multiple in-flight requests, maintain state by request identity rather than arrival order. Responses from different requests must not be appended to whichever conversation happens to be active. This is a protocol problem, not one solved through tokenizer or sampling settings. Keep content assembly and transaction-state changes as separate operations.

Design a client state machine

Define sent, receiving, final received, and failed states, listing valid transitions. Test one response, several responses, no useful text, and two interleaved requests. These are synthetic transport fixtures, not observed language-model results. After final completion, handle subsequent events according to the client contract rather than creating a second finished answer. A connection error must remain distinguishable from a normal empty completion. The fixture should exercise both transitions without depending on a particular answer’s wording or punctuation.

Relate this to TensorRT-LLM

The current deployment guide connects streaming with decoupled configuration and generate_stream. Do not assume ordinary generate is equivalent. Preserve endpoint and model configuration with the test. Batching and response delivery are separate properties; changing one does not establish the other. If an older deployment example follows another backend path, identify that boundary before carrying its HTTP or gRPC assumptions into the LLM API. Protocol compatibility should be established from the chosen path, not inferred from a shared model name.

Accept the complete stream

Keep raw events, assembled text, final condition, and request identity. Test interrupted connections and timeouts separately; readable partial content does not make them successful completions. Do not retry merely because a final payload is empty when the protocol already confirms success. A useful artifact can replay the synthetic state-machine events without a model and then validate them against the chosen backend. It does not promise that every OpenAI streaming client matches every Triton endpoint. Explicit event mapping prevents answer loss at the boundary between model, transport, and client text assembly.

Testing streaming responses from the vLLM API; Errors inside an OpenRouter stream: why HTTP 200 is insufficient.