In this article
A reranker can reorder retrieved candidates but cannot recover evidence the first stage never supplied.
A RAG reranker rescores candidates found by the first retrieval stage. It can improve the short list’s ordering but adds work for each query-passage pair. GPU selection therefore depends on candidate count, text length and the latency budget for the complete request, not only the reranker model’s size.
Separate retrieval and rescoring
Embedding retrieval produces an initial candidate set. A Cross-Encoder or other reranker performs a different evaluation on selected pairs. These stages have different resource profiles. Preserve the original candidate IDs and scores so the ranking change can be understood and tested.
A reranker cannot rank a relevant fragment that retrieval never returned. Check first-stage coverage before attributing every poor answer to rescoring. Use questions with known useful evidence and keep source versions fixed while comparing list depths.
Compare several candidate depths
| Variable | Possible effect |
|---|---|
| Candidate count | More pairs to process |
| Passage length | Different model work per pair |
| Batch size | Changed memory and execution efficiency |
| Simultaneous questions | Competition for the same compute resource |
Choose a few depths suited to your corpus and run the same questions at each level. These are experiment settings, not universal recommended counts. Record actual pair lengths and truncation, because a long candidate may lose the evidence that made it relevant.
Keep an end-to-end latency budget
Measure retrieval, reranking, context assembly and generation separately. A small ranking improvement can be unsuitable if it pushes the entire interaction beyond the application’s waiting limit. Conversely, an offline task may accept a different trade-off from an interactive assistant.
Compare isolated reranking with the real pipeline when stages share a GPU. An individually successful batch does not prove the same latency while generation and indexing compete. Save concurrency, queue behavior and the complete request timeline instead of reporting only model execution time.
Validate ordering and decide placement
Compare results before and after rescoring, preserving cases that improve and cases that get worse. Measure the property users need: relevant evidence near the top, coverage or ordering. A changed numeric score is not by itself proof of a better application result.
The pilot may justify a shared GPU, a separate worker or a simpler pipeline without reranking. For equipment owners and operations funders, the decision should follow observed quality and resource use. No fixed candidate count or measured capacity is assumed before testing.
Official references: sbert.net — quickstart, qdrant.tech — ranking and reranking.
Related guides: BGE rerankers on GPU: testing memory and batching, Qwen rerankers: separating relevance scoring from text generation.