# Local fault injection: interrupting an upstream at a known stream boundary

A local fake upstream for SSE faults before headers, after headers and after content, with outcome classification, bounded retries and a replay artifact.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/provider-midstream-fault-injection-tests/
Language: en

## Build a controllable upstream

Instead of disconnecting a real server at an arbitrary moment, create a local fake upstream with a short predetermined response. It can stop before headers, immediately after headers or after a selected content event. This separates error handling defects from network instability and variable generation duration. Save source bytes, the interruption point and expected classification for each scenario. The fake should neither call a paid model nor copy real user content. Its purpose is reproducing a transport sequence, so a synthetic payload is more useful than nondeterministic generation that cannot reliably be interrupted at the same point again.

## Distinguish three boundaries

Before headers, the service can still construct an ordinary HTTP error response. Once headers are committed, available signalling mechanisms change; once content is forwarded, the client may already have used a partial result. [OpenRouter streaming documentation](https://openrouter.ai/docs/api_reference/streaming) describes errors in an established stream, although that external format is not automatically an upstream requirement for every provider. Keep receipt of headers, first content and a terminal signal as separate test observations. HTTP status alone does not establish generation completeness: a successfully opened connection can end with a damaged or unfinished stream. These observations must not collapse into a single transport success flag.

## Check framing and terminal state

In one case, the fake closes without a final event; in another, it sends a valid error message and completes transport correctly. HTTP/1.1 message completeness depends on the framing described by [RFC 9112](https://www.rfc-editor.org/rfc/rfc9112.html). A complete HTTP message still does not guarantee completion of the application generation. Test both layers independently. Report the last fully parsed SSE event and bytes already forwarded to the client. If an expected error becomes an endless wait, investigate the timeout and reader state machine rather than simply extending the test until the symptom disappears.

## Bound retries and examine the proxy

A proxy can buffer data, change the observed interruption point or make its own upstream retries. The [nginx proxy documentation](https://nginx.org/en/docs/http/ngx_http_proxy_module.html) helps check relevant settings in a particular installation. Test the direct path first, then insert the proxy and compare events. All attempts share a maximum budget and receive separate identifiers. If output has already escaped, automatic replay can duplicate content or mix two generations. The test must detect that result even when the second upstream responds successfully. Explicitly test fallback before visible content as well; do not assume safety from an HTTP status alone.

## Preserve the reproducible scenario

The artifact includes the fake version, byte fixture, failure offset, proxy configuration, actual attempts and final client status. See [streaming errors](/journal/openrouter-stream-errors/) and [inference routing fallback](/journal/inference-routing-fallback/). Bound the suite by duration and scenario count, and confine intentional corruption to the test environment. Passing confirms handling of selected interruption points, rather than resilience to every network failure or correctness of a remote engine. Preserve a minimal case for unexplained differences and separately check whether the test harness changed event ordering. That is more valuable than a large report counting arbitrary restarts.