In this article
A controlled distractor test retains the original relevant evidence and answer target while changing only the added context.
RAG noise tests should change the surrounding context while keeping the question and answer target fixed. This reveals whether irrelevant or misleading passages alter the answer, instead of confusing robustness with a different retrieval task or a newly defined reference answer.
Define the supported answer first
Choose a question with a clear source-backed answer and preserve the relevant evidence. The Ragas metric catalog distinguishes noise sensitivity from other evaluation dimensions. An incomplete or disputed reference makes the experiment ambiguous: a changed answer might be a correction rather than noise-induced failure. Review the reference before constructing distractors, and retain the exact source revision that establishes the intended answer.
Create matched context conditions
Use an equipment question asking which inspection applies to the current revision. The clean condition contains the relevant procedure. Another condition adds an unrelated service note; another adds a superseded procedure with similar vocabulary. Keep the current procedure present in every condition. These additions test different failure mechanisms: irrelevant distraction and confusion between superficially relevant but incompatible evidence. They should not be merged without retaining their separate labels.
| Condition | Added material | Diagnostic focus |
|---|---|---|
| Clean context | None | Baseline answer behavior |
| Unrelated addition | Other equipment note | Resistance to irrelevant context |
| Conflicting revision | Superseded procedure | Selection of applicable evidence |
Preserve the controlled variables
The noise-sensitivity documentation describes metric-specific inputs and interpretation. Keep the question, reference, generator configuration and original evidence unchanged while varying the addition. Record passage order separately because moving the relevant passage can change behavior. If the experiment also removes the relevant source, it becomes an answerability test; retain that as a different condition rather than calling every failure noise sensitivity.
Inspect claims rather than fluency
Compare whether the answer retains the applicable procedure, adopts a distractor’s obsolete instruction or adds unsupported details. A polished response can still select the wrong revision. Conversely, a shorter answer may remain correct. Model-based evaluation can help organize claims, but the primary implementation does not turn its judgments into verified ground truth. Review cases involving technical negation, version applicability and partially supported instructions manually.
Report failures by distractor mechanism
Present the matched contexts and answer changes, not only a single robustness score. Separate irrelevant text, overlapping vocabulary and conflicting revisions. Explain whether errors arose from source selection, unsupported synthesis or loss of the answer entirely. The result can guide metadata filtering, context construction or generation instructions. It cannot establish universal resistance to arbitrary documents, and it should not disguise a change in reference answer as an improvement in robustness.
Related reading: Reranking in RAG: how many candidates should you rescore?; A local RAG server: placing components around the GPU.