In this article
A high aggregate embedding score can conceal regressions on the particular retrieval task or language required by the application.
MTEB and MMTEB scores become useful only after selecting tasks and languages that match the intended retrieval workload. A leaderboard average cannot establish which embedder will find the right maintenance paragraph, multilingual policy clause or code example in a particular collection.
Separate the benchmark from the deployment task
The MTEB paper organizes multiple embedding tasks, including tasks that are not document retrieval. Strong classification or clustering performance is useful evidence about those tasks, but it does not replace retrieval evaluation. Define whether the application needs query-to-passage search, symmetric similarity or another relationship before comparing model summaries or choosing a convenient overall score.
Inspect multilingual coverage explicitly
The MMTEB paper expands multilingual evaluation. Language coverage still does not make every task representative of every domain. A Russian technical query containing an English part number differs from ordinary monolingual sentence similarity. Record document language, query language and the relationship between them. Cross-language retrieval deserves its own slice rather than disappearing into a multilingual average.
Build a selection matrix
Consider a bilingual equipment library. Users ask short troubleshooting questions, look up exact identifiers and compare descriptions. Those are different retrieval needs. Use benchmark tasks to identify plausible candidates, then test each candidate on held-out questions from the library. Keep exact-identifier cases visible because semantic relevance alone may not preserve a product code.
| Workload slice | Selection question |
|---|---|
| Short question to paragraph | Is an asymmetric retrieval task represented? |
| Russian query to English manual | Is the language direction evaluated? |
| Exact component identifier | Does the local test preserve literal distinctions? |
| Similar descriptions | Is symmetric similarity actually required? |
Pin the runner and model contract
The current MTEB repository contains tasks and evaluation interfaces. Save its revision, selected tasks, dataset versions and model revision. Query prompts, pooling, normalization and truncation belong to the model configuration. Comparing numbers produced with different task revisions or preprocessing can mistake an evaluation change for a model improvement, even when the displayed model names remain unchanged.
Read aggregation before using a ranking
An average compresses different tasks, languages and dataset sizes into a summary. Inspect how that summary is constructed and which results are missing. A model can improve the average while regressing on the language or document type that matters locally. Report the relevant task results beside the aggregate, and keep operational constraints separate from quality measurements.
Make the shortlist auditable
The final decision should name the benchmark evidence, local fixture and unresolved gaps for every candidate. Do not turn vendor-published scores into locally measured results. A reasonable shortlist may retain a smaller model with adequate local retrieval or a larger model that materially improves a difficult slice. Either conclusion requires evidence tied to that workload, rather than a universal recommendation inferred from leaderboard position.
Related reading: Evaluating embeddings for Russian language retrieval; Batch text embeddings on your own GPU.