Local video search: join transcript evidence with visual keyframes

Local video search should join transcript spans and selected frames through source time, while preserving each modality’s evidence separately. A spoken reference and a visually similar frame are not automatically the same event, and combining them carelessly can create an answer the recording never supported.

Create a shared temporal reference

The PyAV time reference explains timestamp interpretation. Keep video presentation times and audio positions mapped to the original recording. Any trimming, resampling or frame-rate conversion needs a stored transformation. A common filename does not establish temporal alignment. Check the actual stream relationship before attaching a transcript paragraph to a frame selected from another processed version of the same media file.

Keep speech and visual records distinct

The WhisperX project supplies aligned speech workflows, while the official Docling video example illustrates video ingestion. Store transcript spans with text and timing, and frames with image identity and presentation time. A combined search layer can retrieve either or both. Preserve which modality supplied a match so the answer does not imply that a spoken claim was visibly demonstrated or that visible text was actually spoken.

Test a delayed visual explanation

Use a software demonstration where the presenter announces a setting before opening its dialog, then later describes a different setting while the old dialog remains visible. Ask where the first setting is changed. A naive nearest-time join can pair the second explanation with the first image. Review the surrounding interval and retrieve enough context to distinguish announcement, action and later commentary.

Evidence record	What it establishes
Transcript span	Words recognized in a recording interval
Selected frame	Visual state at a presentation time
Temporal link	Co-occurrence under a defined window
Reviewed event	Relationship between explanation and action

Construct retrieval candidates by modality

A text query may match transcript wording, OCR-derived labels or a visual representation. Keep candidate provenance and avoid treating their scores as automatically comparable. Combine results through an explicit policy and inspect whether the union actually contains the answer-bearing event. When a frame lacks the needed detail, retrieving nearby speech does not repair the missing image unless that speech independently establishes the fact.

Validate source-grounded answers

Review questions about spoken statements, visible actions and relationships between them as separate slices. Resolve results to playback and the original frame, retaining uncertain timing or recognition. Report missed events and incorrect cross-modal joins separately. The deliverable is a searchable evidence timeline with recoverable source links, not a claim that every visual event can be inferred from speech or that temporal proximity proves causal or semantic agreement.