Nemotron 3.5 Rerank: interpret scores before choosing a relevance threshold

Nemotron 3.5 Rerank scores should be interpreted under the checkpoint’s output contract before setting any rejection threshold. Ranking candidates and deciding that no candidate is adequate are separate tasks; a useful ordering score is not automatically a calibrated probability of relevance.

Separate candidate retrieval from reranking

The NVIDIA family announcement distinguishes embedding retrieval from reranking. A reranker evaluates supplied query-document pairs; it cannot recover a relevant page absent from its input candidate set. Keep initial candidate coverage visible when evaluating the reranker. Otherwise an empty answer may be blamed on its threshold even though the retrieval stage never supplied the necessary evidence.

Read the score contract directly

The official Nemotron 3.5 Rerank card describes a relevance logit for text queries paired with documents that can contain text, page images or both. The published checkpoint configuration identifies the classification architecture. A wrapper may apply its own output transformation. Record whether stored values are raw model outputs or transformed values before comparing integrations or reusing a threshold from another service.

Use ranking and rejection fixtures

Build a set containing a relevant current procedure, a related obsolete procedure and an unrelated note. First inspect their ordering for the same question. Then add questions whose answers are absent from every candidate. These test a different decision: whether the pipeline can refrain from treating its best available candidate as sufficient evidence. Do not derive that decision merely from the fact that a top-ranked candidate exists.

Fixture	Question answered
Relevant and unrelated candidates	Does ordering separate obvious relevance?
Current and obsolete procedures	Does the model respect applicability?
No supporting candidate	Does rejection avoid unsupported answers?

Fit a threshold to the actual workload

Use development judgments to choose a decision rule, then inspect held-out false acceptance and false rejection. Keep text-only and image-containing documents separately visible, since their preprocessing and evidence differ. A threshold selected on easy negatives can fail on plausible but outdated passages. Preserve the candidate distribution and query types alongside the threshold; changing either may invalidate its interpretation even if the checkpoint remains unchanged.

Version the score-to-decision boundary

The operational artifact is a contract containing checkpoint revision, processor, score transformation, decision rule and reviewed examples. After a wrapper or model update, rerun both ranking and rejection fixtures. Report errors as missing candidates, wrong ordering or incorrect rejection rather than one combined quality number. This keeps a documented reranker role distinct from unverified confidence claims and makes the downstream answer policy inspectable.