# Video retrieval timestamps: use PTS rather than frame-number arithmetic

Variable-frame-rate video should be indexed with presentation timestamps and time bases rather than assuming that frame number divided by nominal frame rate gives exact time.

Author: Retrivium team
Updated: 2026-11-03
Canonical URL: https://retrivium.ai/journal/video-vfr-pts-frame-indexing/
Language: en

Variable-frame-rate video should be indexed with presentation timestamps and time bases rather than assuming that frame number divided by nominal frame rate gives exact time. Otherwise a retrieved frame can point to the wrong spoken explanation, especially when capture timing is irregular or preprocessing changes cadence.

## Preserve timestamp meaning during decoding

The [PyAV time documentation](https://pyav.basswood.io/docs/stable/api/time.html) explains timestamps and time bases. A timestamp value needs its associated time base to become a meaningful time position. Keep original stream identity and timestamp metadata with selected frames. Do not treat packet decode order, frame presentation order and a sequential frame counter as interchangeable evidence, particularly when the container or codec reorders data during decoding.

## Separate nominal rate from actual timing

The [PySceneDetect backend documentation](https://www.scenedetect.com/docs/head/api/backends.html) explicitly warns about approximate frame numbering for variable-frame-rate inputs. A nominal rate is useful metadata but cannot reconstruct every irregular presentation interval. Retain the presentation time of the actual decoded frame. If an application needs a frame number for display, label how it was derived rather than presenting it as a universally exact location in the source.

## Test a screen recording with pauses

Choose a screen recording with irregular frame updates and a narrated action. Select the frame showing a changed setting, then map it back to the corresponding narration. Compare timestamp-based positioning against a nominal-rate calculation. This is a diagnostic fixture, not a claimed measured drift: the point is to expose whether the pipeline relies on an assumption the input does not satisfy.

| Stored field | Purpose |
| --- | --- |
| Presentation timestamp | Position of the decoded frame |
| Time base | Interpretation of timestamp units |
| Stream identifier | Distinguishes video from other tracks |
| Derived frame label | Convenient display with explicit assumptions |

## Account for seeking and conversion

The [FFmpeg documentation](https://ffmpeg.org/ffmpeg-doc.html) describes seeking and timestamp-related options. Seeking can involve decoding from a nearby point rather than retrieving an arbitrary exact image instantly. A conversion to constant frame rate can duplicate or discard frames. Record such transformations and preserve the source-time relationship, so an index built from converted video does not imply unchanged original frame identity or timing.

## Validate cross-modal evidence

Resolve selected frames and transcript spans to the original recording and inspect their agreement around visible actions. Test early, late and irregular portions rather than a single convenient frame. Keep timestamp mapping with the index revision. This makes video citations reproducible and prevents a correct visual match from being paired with unrelated speech merely because an approximate frame counter was mistaken for authoritative source time.

Related reading: [A local RAG server: placing components around the GPU](/journal/rag-server-local-architecture/); [Long audio with Whisper: segments, queues and durable results](/journal/whisper-long-audio/).