ColPali similarity maps: model focus is not an OCR citation box

ColPali similarity maps explain which image patches match query tokens, but they do not establish an OCR quotation or a verified citation box. Use them to investigate page retrieval while keeping source identity, extracted wording and coordinate-based evidence as separate records.

Understand what the map represents

The ColPali paper describes visual document retrieval through late interaction between query and page representations. The author repository includes similarity-map tools for inspecting these interactions. A bright region indicates a model similarity contribution under that representation. It does not prove that every character inside the region was read correctly or that surrounding text supports a proposed answer.

Preserve the page behind the visualization

The ColPali checkpoint card identifies a specific page-image retrieval model. Save its revision, original page identifier and rendering settings alongside the visualization. Resizing can move apparent coordinates, and a screenshot of a heatmap is insufficient to reconstruct the underlying page. Keep the plain page image available so reviewers can inspect evidence without the overlay obscuring labels.

Use a deliberately ambiguous diagram

Consider a wiring page containing a connector diagram, a warning box and a repeated product heading. Ask which connector carries the sensor signal. A useful inspection separates contributions associated with the connector word from contributions associated with generic product vocabulary. The warning box can attract similarity while containing no connector answer; this is an interpretation task, not proof of model reasoning.

Artifact	What it supports
Ranked page identifier	Location of a retrieved candidate
Query-token similarity map	Inspection of representation contributions
OCR text with coordinates	Recoverable wording and geometric location
Human-checked citation	Support for a specific answer claim

Connect maps to a citation workflow

After selecting the page, recover the relevant wording or inspect the diagram directly. If a citation requires a text box, obtain that box from a coordinate-preserving extraction process and verify its contents. Do not convert the brightest patch into a quotation automatically. A patch may span several words, a graphical symbol or empty space near a meaningful feature.

Diagnose disagreement between artifacts

If the right page ranks highly but the highlighted region is wrong, inspect query-token contributions individually. If a plausible highlight accompanies incorrect OCR, retain the image as primary evidence and flag the transcription. If the map focuses on a repeated header across unrelated pages, evaluate whether document layout or query phrasing drives the match instead of answer-bearing content.

Report the actual strength of evidence

An inspection record should contain the question, page, map, recovered evidence and a conclusion about what remains uncertain. Similarity maps help locate retrieval shortcuts and guide manual review. They should remain explanatory artifacts rather than an automatic authority for claims, bounding boxes or calibrated confidence. This distinction makes visual retrieval useful without attaching stronger evidential meaning than its output provides.