In this article
A video presentation timestamp needs its time base; a nominal frame rate cannot reconstruct irregular frame intervals.
Variable-frame-rate video should be indexed with presentation timestamps and time bases rather than assuming that frame number divided by nominal frame rate gives exact time. Otherwise a retrieved frame can point to the wrong spoken explanation, especially when capture timing is irregular or preprocessing changes cadence.
Preserve timestamp meaning during decoding
The PyAV time documentation explains timestamps and time bases. A timestamp value needs its associated time base to become a meaningful time position. Keep original stream identity and timestamp metadata with selected frames. Do not treat packet decode order, frame presentation order and a sequential frame counter as interchangeable evidence, particularly when the container or codec reorders data during decoding.
Separate nominal rate from actual timing
The PySceneDetect backend documentation explicitly warns about approximate frame numbering for variable-frame-rate inputs. A nominal rate is useful metadata but cannot reconstruct every irregular presentation interval. Retain the presentation time of the actual decoded frame. If an application needs a frame number for display, label how it was derived rather than presenting it as a universally exact location in the source.
Test a screen recording with pauses
Choose a screen recording with irregular frame updates and a narrated action. Select the frame showing a changed setting, then map it back to the corresponding narration. Compare timestamp-based positioning against a nominal-rate calculation. This is a diagnostic fixture, not a claimed measured drift: the point is to expose whether the pipeline relies on an assumption the input does not satisfy.
| Stored field | Purpose |
|---|---|
| Presentation timestamp | Position of the decoded frame |
| Time base | Interpretation of timestamp units |
| Stream identifier | Distinguishes video from other tracks |
| Derived frame label | Convenient display with explicit assumptions |
Account for seeking and conversion
The FFmpeg documentation describes seeking and timestamp-related options. Seeking can involve decoding from a nearby point rather than retrieving an arbitrary exact image instantly. A conversion to constant frame rate can duplicate or discard frames. Record such transformations and preserve the source-time relationship, so an index built from converted video does not imply unchanged original frame identity or timing.
Validate cross-modal evidence
Resolve selected frames and transcript spans to the original recording and inspect their agreement around visible actions. Test early, late and irregular portions rather than a single convenient frame. Keep timestamp mapping with the index revision. This makes video citations reproducible and prevents a correct visual match from being paired with unrelated speech merely because an approximate frame counter was mistaken for authoritative source time.
Related reading: A local RAG server: placing components around the GPU; Long audio with Whisper: segments, queues and durable results.