Synthetic RAG test sets: keep document leakage out of the holdout

Synthetic RAG questions should be generated after separating source documents into development and holdout groups. Splitting generated questions alone can leave nearly identical passages on both sides, making a tuned retrieval pipeline appear to generalize when it is being tested against familiar evidence.

Split the source before generating questions

The official Ragas testset workflow generates evaluation material from documents. Before applying it, group duplicate files, revisions and fragments derived from the same original. Assign those groups together. Otherwise a question about a paragraph in one manual edition can enter the holdout while the development set contains its nearly identical counterpart. The grouping rule belongs in the dataset manifest, alongside document hashes and version relationships.

Preserve the generator’s provenance

Record the generator model, instructions, source document identifiers and generated reference answer. The Ragas implementation provides the workflow machinery, but generated material still requires editorial inspection. A question may expose the exact wording of its answer, rely on missing context or combine incompatible document versions. Rejecting such cases should follow a stated rule rather than whether a preferred retriever happens to answer them correctly.

Build a leakage inspection table

Imagine a service manual, an updated edition and an HTML export of the older edition. A random file split treats these as independent documents. A source-family split recognizes their shared evidence. Keep a small table showing the proposed assignment and why related artifacts remain together. This is an illustrative data-control fixture, not a claim about measured benchmark inflation.

Artifact	Source relationship	Split handling
Original manual	Source family	Assign as a group
Updated manual	Related revision	Review overlap before assignment
HTML export	Duplicate representation	Keep with its original

Separate questions from relevance judgments

The BEIR data organization distinguishes corpus, queries and relevance judgments. Apply that separation to synthetic tests. A generated answer does not automatically identify every relevant passage, and a question’s source passage is not necessarily the only valid evidence. Manually review ambiguous cases and retain unjudged status where appropriate. Keep the untouched generated output so later corrections remain traceable rather than silently replacing the original artifact.

Protect the holdout during tuning

Use development questions to select chunking, overlap and retrieval settings. Consult the holdout only for a declared evaluation, then version any subsequent reuse. A disappointing score should trigger error analysis rather than another unrecorded tuning cycle against the same holdout. The final report should show source grouping, rejection reasons and residual uncertainty. This provides evidence of generalization boundaries without presenting synthetic questions as an independent representation of real user demand.