In this analysis
Report fresh-start and sustained-use observations separately before assigning a cause to a changing device result.
Pipette’s on-device measurement framing makes device state part of reproducibility, so repeated runs need a declared lifecycle rather than an unexplained average.
The systems perspective
The platform announcement treats the deployed configuration as the measurement subject. On a physical device, that configuration includes more than the downloaded model. The state in which a run begins can differ from the state after sustained work, and the distinction can matter to how an observation should be interpreted.
Earlier edge context
The earlier hybrid release and compact release document the preceding on-device direction. They do not supply a measured thermal effect for a later experiment. This article derives a protocol question from the systems framing; it does not claim that a particular device throttled by an unreported amount.
An honest retrospective separates that reasoning from results. A plausible physical mechanism is a reason to control or record a condition, not permission to invent its magnitude or assume that it dominated an observation.
A repeated-demo example
Imagine running the same demonstration at the beginning and end of a long session. The model file and prompt can remain unchanged while background activity and device state differ. If the later run is slower, the timing alone cannot identify which changed condition caused the difference.
Record the run order and the relevant state available to the operator. If a cooling interval or restart is part of the protocol, preserve that fact too. Otherwise the comparison silently combines a fresh-start condition with a sustained-use condition while presenting both as repetitions of one experiment.
Averaging can hide the question
An average across mixed lifecycle stages may be mathematically valid yet operationally confusing. A user interested in a first interaction and a user interested in sustained work are asking different questions. Combining those states without labels removes the distinction needed to answer either question precisely.
Keep individual runs available before summarizing. A trend can motivate further investigation, but it does not identify its cause by itself. Temperature, power policy, background work and preparation state are candidate explanations to distinguish, rather than interchangeable names for any observed slowdown.
The retrospective contribution
A system-oriented benchmark is useful when its protocol makes those conditions inspectable. The platform’s significance is not a guarantee that every comparison is automatically fair, but a framing in which the operator can state what was held fixed and what changed.
The conclusion is to report the lifecycle being measured. Fresh-start behaviour and repeated sustained behaviour can both matter; keeping them separate produces stronger evidence than a single number whose underlying device states have been mixed together.
Further analysis: Reproducible inference stacks: preserving a GPU pilot’s result; GPU hosting for multimodal models: count more than text.