# SSE and UTF-8: a network chunk is not a JSON event

Understand why network reads are not SSE events and preserve UTF-8 across chunks with incremental decoding, framing, fragmentation fixtures and bounded buffers.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/provider-sse-utf8-chunk-boundaries/
Language: en

A server can produce a valid stream that breaks in a client because processing uses the wrong boundary. One network read may contain half a character, several SSE events or part of a data line. Parsing every received chunk as standalone JSON makes failures depend on arbitrary network fragmentation. Reliable validation uses successive layers and preserves state between reads.

## Separate bytes from text

The [SSE standard](https://html.spec.whatwg.org/multipage/server-sent-events.html) specifies UTF-8 for event streams. This does not mean each network fragment contains complete characters. A multibyte sequence can continue in the next read, so decoding must retain its unfinished remainder.

[TextDecoder.decode](https://developer.mozilla.org/en-US/docs/Web/API/TextDecoder/decode) documents streaming mode, where subsequent calls supply additional data. Finishing decoding is also a separate step. Constructing a fresh decoder for every chunk loses state even when all original bytes arrive correctly and in order.

## Frame events after decoding

Decoded text enters an SSE parser that assembles lines and events under the format rules. Only a complete data payload reaches JSON parsing. Do not assume one read equals one event or one line always contains a complete object.

Streams can contain comments and several data lines belonging to one event. Handle them according to the format rather than a convenient regular expression. A custom parser needs explicit unfinished-line and current-event state, with limits on accumulation for malformed input.

## Create a Unicode fixture

Use a short synthetic event containing Latin text, a Cyrillic character and an emoji. Save original bytes and expected final text. Split it at selected positions, including inside a multibyte character, inside a data prefix and between neighboring events.

Every partition must produce the same application-event sequence. This verifies independence from transport fragmentation. Begin with a bounded cut set; enumerating every partition is unnecessary when it greatly expands runtime without helping expose a specific implementation fault.

## Preserve multiple events

The [MDN Streams API guide](https://developer.mozilla.org/en-US/docs/Web/API/Streams_API/Concepts) explains that chunks vary in size and pass through read queues. One large read can therefore contain several complete events. Extract each and preserve the remainder for the next read.

Check event count, order and concatenated content. A successful final JSON parse does not establish that preceding events survived. Add an empty read, normal closure and truncation with an unfinished line, retaining their different outcomes.

## Limit conclusions to transport

Passing a fixture validates decoding and framing on that set. It does not establish tool-argument correctness, finish_reason meaning or generation quality. Those layers should receive correctly framed events and maintain their own expectations so one error cannot hide another.

The [OpenRouter streaming client](/journal/openrouter-streaming-client/) covers incremental reading, while [vLLM streaming tests](/journal/vllm-openai-streaming-test/) provide a first server example. Bytes, text, SSE and JSON remain distinct boundaries regardless of engine or router. Keep the fixture's input bytes, cut positions, expected events and parser version together so a later change can reproduce the same evidence without contacting a real model.