ColBERT in Qdrant: preserve token vectors and MaxSim semantics

A ColBERT representation is a matrix of token vectors, not a longer ordinary embedding to average before searching. In Qdrant, preserve that representation and its MaxSim scoring contract while separating first-stage candidates from late-interaction reranking.

Preserve the matrix boundary

The Qdrant vector reference documents multivectors and the MaxSim comparator. A document can contribute several vectors, and the query also has a structured representation. Averaging document tokens into one vector changes the retrieval method. It may produce a useful separate baseline, but it cannot be described as the same ColBERT scorer merely because both representations originate from one checkpoint.

Separate candidate generation from scoring

The hybrid-query documentation provides a prefetch stage whose candidates feed a subsequent query. Use a separate dense representation for a first-stage experiment when that is the intended design, then apply the multivector scorer to its candidates. Preserve candidate IDs before reranking. A relevant document missing at this boundary cannot be recovered by a perfect second-stage scorer, so candidate recall and reranking quality need different measurements.

Keep the model-specific preprocessing

The author ColBERT implementation establishes token-level late interaction, including its own query and document processing. Do not replace those processors with an arbitrary sentence tokenizer or assume that equal vector dimensions establish compatibility. Record checkpoint revision, tokenization, padding and retained token positions. An array with the correct shape can still represent the wrong task when query and document roles were swapped.

Test with an explainable document pair

Build an illustrative fixture with a short query and two passages: one shares a rare product identifier, while the other repeats general topic words. Preserve their actual text and model outputs. Inspect whether changing token retention alters the ordering. This is a diagnostic experiment, not a promised winner. Add a longer passage afterward to expose truncation effects; otherwise an apparently poor score may simply concern evidence never encoded.

Boundary	Artifact to retain
Encoding	Token matrices and masks
Prefetch	Candidate document IDs
Reranking	Final scores and ordering

Account for index and scoring work

Token matrices contain more vector elements than a single pooled document representation. Their actual footprint depends on retained tokens and the indexing arrangement. Measure those quantities rather than multiplying one embedding dimension by document count and declaring a complete budget. If graph indexing is disabled for a reranking-only field in the selected configuration, retain that decision explicitly; it changes the field’s role rather than removing the need to store and score its vectors.

Accept the complete retrieval chain

Compare candidate coverage first and final relevance afterward on the same judgments. Preserve a small exact scorer fixture when changing client or engine versions. Document whether a result came from a full multivector search or a bounded reranking stage. That distinction allows a later operator to explain a missed passage without incorrectly blaming MaxSim for a document that never reached it.