Mixed PDF OCR: avoid duplicating an existing text layer

THE QUICK TAKE

Native text and OCR output can describe the same page content, so longer extracted text may indicate duplication rather than better evidence.

Mixed PDFs require an OCR policy that distinguishes existing digital text from text contained only in images. Adding another recognized layer indiscriminately can duplicate evidence, disturb reading order and make a repeated phrase appear more important than it was in the original document.

Inspect text and images separately

Use the official text extraction recipes to establish what the original PDF already exposes. A scanned page with a hidden OCR layer differs from a page containing no text at all. A digital page can also contain an image with essential labels. Inspect representative pages rather than classifying the whole file from whether its first page returns a nonempty string.

Choose full or partial recognition deliberately

The Page reference documents get_textpage_ocr and its options, including full-page versus partial OCR behavior. The returned TextPage representation must be used appropriately in subsequent extraction. Preserve the distinction between original extraction and OCR-derived output. Do not assume that invoking OCR permanently replaces the PDF’s text or that later extraction automatically uses the newly created text representation.

Build a mixed-content fixture

Choose a page with a digital heading, a scanned table and a pre-existing hidden transcription. Compare the original extraction, the chosen OCR path and the page image. The table’s labels should become available without duplicating the heading or accepting a stale hidden transcription as independent corroboration. Keep a human-reviewed expected phrase list to expose both omissions and repeated content.

Page condition	Main risk
Digital text only	Unnecessary duplicate recognition
Image text without transcription	Missing answer-bearing labels
Hidden OCR already present	Competing or repeated transcripts
Mixed text and images	Broken ordering between evidence regions

Reconcile text without erasing evidence

Do not deduplicate every repeated phrase across a document: warnings and table headers can legitimately recur. Compare overlapping content using page location and extraction provenance. If native text and OCR disagree, preserve both for inspection instead of selecting the longer string automatically. OCR can misread an identifier while the native layer is correct, or the hidden layer can be outdated while the image is authoritative.

Validate downstream behavior

Inspect whether a retrieval query returns duplicated chunks, an incorrect transcription or a passage assembled in the wrong order. Save the source page, OCR configuration and selected text representation with the test. The acceptance criterion is usable evidence with traceable provenance, not a larger character count. A successful OCR operation does not itself establish that normalization, chunking and citation coordinates now refer to a single coherent version of the page.