In this analysis
Attach a runtime revision to each measurement before attributing an upgrade result to model quality or execution performance.
Pipette’s deployed-system perspective makes a runtime revision a benchmark variable, rather than an invisible detail behind an unchanged model name.
The platform’s unit of comparison
The announcement frames measurements around deployed configurations. A runtime participates in that configuration because it selects execution paths and interprets artifacts. Two results labelled with the same model can therefore describe different systems when their runtime revisions differ, even if the task and hardware appear unchanged.
Earlier releases provide context
The earlier hybrid model and compact model establish the preceding deployment programme. They do not provide a fixed runtime control for every later experiment. Their role is to locate the measurement platform historically, not to make any current implementation the timeless reference for older observations.
The distinction prevents a common attribution error: a changed system result is described as improved model quality when the weights never changed. Execution improvements can be valuable, but their cause should remain attached to the component that supplied them.
A revision comparison example
Imagine repeating a local extraction task after updating only the runtime. The answer may remain the same while startup, memory use or request handling changes. A model-only experiment log would hide the changed component and leave the new observation without a defensible causal explanation.
Keep the previous runnable configuration, or record exactly which part could not be preserved. The point is not to forbid upgrades. It is to make a comparison between old and new paths possible without reconstructing missing versions from memory after an unexpected result.
Output agreement is another axis
An execution change can also alter numerical behaviour or formatting. A performance comparison should retain the task’s acceptance condition rather than assuming that successful completion implies equivalent outputs. Conversely, different text is not automatically evidence of a regression when the task permits several correct answers.
Use the same source fixture and the same acceptance rule while changing the selected runtime. If another variable must change too, describe the experiment as a configuration comparison. That is still useful evidence, but it supports a different causal claim from an isolated runtime comparison.
The retrospective lesson
The platform makes execution identity part of measurement interpretation. This helps separate the quality of an artifact from the behaviour of the system that executes it, without pretending those layers are independent in every practical outcome.
A defensible result therefore records the runtime alongside model, representation and task. Preserving that identity turns an upgrade from an unexplained before-and-after observation into a comparison whose limits can be understood and whose useful conclusion can be attributed to the right layer.
Further analysis: Reproducible inference stacks: preserving a GPU pilot’s result; GPU hosting for multimodal models: count more than text.