BEIR qrels: an unjudged passage is not a verified negative

A missing BEIR relevance judgment does not establish that a passage is a verified negative. Preserve the distinction between unjudged material and explicitly reviewed irrelevant material when constructing retrieval tests, especially when comparing a new retriever against an incomplete judgment pool.

Keep the benchmark objects separate

The BEIR repository separates the corpus, queries and relevance judgments, commonly called qrels. Corpus identifiers connect passages to judgments; query identifiers connect questions to their evaluation targets. Joining these objects through row positions or regenerated identifiers can silently change the task. Treat identifiers as stable keys and check that every referenced document remains present after preprocessing.

Read what an absent judgment means

The BEIR paper evaluates heterogeneous retrieval tasks rather than declaring every corpus item exhaustively judged. Evaluation software necessarily assigns metric consequences to missing judgments, but those consequences are not new human assessments. A retriever can surface a useful passage outside an existing judgment pool. That result requires review before calling it either a retrieval improvement or a false positive.

Create a judgment-aware fixture

Consider a maintenance question with a judged relevant procedure, a reviewed unrelated product sheet and a newly discovered troubleshooting note. The note is initially unjudged. Preserve that status in the inspection table rather than assigning a manually invented negative label. If reviewers later judge it relevant, version the qrels and rerun both systems against the same updated set.

Passage	Judgment status	Interpretation
Procedure	Relevant	Known supporting evidence
Other product sheet	Reviewed irrelevant	Explicit negative example
Troubleshooting note	Unjudged	Requires assessment before interpretation

Audit evaluation inputs before scores

The official evaluation implementation accepts qrels and retrieval results keyed by identifiers. Inspect a sample of those joins before reading an aggregate metric. Removing documents without updating judgments changes achievable recall. Filtering queries that lack convenient answers changes the evaluated population. Either transformation needs an explicit dataset version and explanation, not merely a cleaner-looking result table.

Separate metric computation from adjudication

Keep the original benchmark score for reproducibility, then maintain a separate reviewed-error analysis. Do not silently rewrite judgments because a preferred retriever returned plausible text. Have reviewers inspect relevance against the question and source, ideally without seeing which system retrieved it. Distinguish graded relevance, binary relevance and answer sufficiency according to the chosen task definition.

Publish the evaluation boundary

A useful report identifies corpus revision, query split, qrels revision, document filtering and the treatment of unjudged results. Include examples where systems disagree and explain which remain unresolved. This makes benchmark numbers interpretable without pretending the judgment pool is exhaustive. The objective is a stable comparison and a defensible error analysis, rather than converting every missing label into certainty about the underlying document.