In this article
A reranker returns relevance scores rather than document embeddings; its usable batch depends on runtime inputs as well as weights.
For a BGE reranker on GPU, choose the exact model and measure memory and latency on query-document pairs of representative lengths. Weight-file size does not determine the largest usable batch. Runtime data and long passages change the workload, so test them explicitly before describing the card’s capacity.
Establish the model’s role
The BAAI/bge-reranker-v2-m3 card describes pairwise relevance scoring. Its output is a score for ranking, not a document embedding. Loading the reranker does not replace building a vector index or retrieving the initial candidates. Preserve query and candidate IDs throughout the pipeline.
Record the model revision, library version and numeric format. The official FlagEmbedding project provides tools for BGE application and evaluation. Changing precision can change resource use and results, so repeat quality checks rather than assuming a faster configuration is equivalent.
Prepare different input lengths
| Group | Changed property | Measurement |
|---|---|---|
| Short pairs | Small document input | Request overhead |
| Typical pairs | Ordinary corpus passages | Working latency and memory |
| Long pairs | Length-distribution tail | Memory peak and truncation |
| Mixed batch | Different lengths together | Batch-packing behavior |
Use permitted examples with known useful passages. Keep the preparation rules and relevance instruction consistent. An input truncated before its important evidence creates a different ranking problem; save that fact rather than recording only the original document length.
Increase batch size gradually
Start with a small batch that passes the control cases, then change only the number of pairs. Record peak GPU memory, completion time and errors. Repeat the test with long and mixed inputs before selecting a working setting. A short-input maximum should not become an unconditional service limit.
Then add representative simultaneous queries. Several independently valid batches can compete for the same device, particularly when generation shares the GPU. Preserve the entire pipeline latency so reranker execution time is not mistaken for what the user experiences.
Check usefulness and state the boundary
Compare candidate order before and after scoring. Keep improvements and regressions, and verify that relevant candidates existed before reranking. A plausible score or changed order is not proof of better retrieval. Use the same labeled questions when changing numeric settings or model versions.
The final profile should identify model, pair lengths, batch, concurrency, memory and observed quality. For hardware owners, that is a concrete possible role for a card.
Official references: huggingface.co — bge reranker v2 m3, github.com — FlagEmbedding.
Related guides: Reranking in RAG: how many candidates should you rescore?, Qwen rerankers: separating relevance scoring from text generation.