How forgetting became the real test of continual learning

The sleep-framework discussion makes forgetting visible as an evaluation problem. Retrospectively, the challenge changed when learning stopped being a single training event and became a repeated sequence of updates.

The dated path

• 2025-06-12: Self-Adapting Language Models.

• 2025-12-31: Nested Learning: The Illusion of Deep Learning Architectures.

• 2026-10-27: Google explains the sleep framework for continual learning.

Why a fresh-task score became insufficient

A new checkpoint can improve on recent examples while quietly weakening an older skill. If evaluation follows the incoming data distribution, the regression may remain invisible until a different user returns to an earlier task. Our reading of the historical progression is that sequential learning requires a second axis: performance across time. The test must ask what the model retained after each transition, rather than treating the final checkpoint as though it had no history.

A useful experimental comparison

A practical experiment can hold the task sequence fixed and evaluate the same old-task set after every adaptation stage. Separate genuine retention from repeated exposure to those old test answers. Also keep an unchanged-model baseline, a retrieval-assisted baseline and an update-budget record. These comparisons answer different causal questions. They distinguish the value of additional training from the value of simply making relevant information available when the model responds, without pretending that one mechanism replaces every other mechanism.

The failure hidden by an average

Suppose an update helps a common writing task but harms a less frequent structured extraction task. An aggregate score weighted by current traffic might rise despite a serious regression for the affected users. Reporting performance by task family and by update stage makes that tradeoff inspectable. A retention claim should therefore state which abilities were tested and which were outside scope. The resulting evaluation is more demanding, but it measures the promise that persistent learning actually makes.

What would change the retrospective conclusion

The strongest evidence would show durable improvement on new tasks, bounded regressions on old ones and a repeatable relationship between consolidation cost and benefit. If improvements disappear when tasks are reordered, the ordering is part of the result rather than an incidental detail. For serving infrastructure, this argues for checkpoint-level evaluation records and controlled promotion of updates. It does not establish that a research result already generalizes to every deployed language model or every user population. A deployment can also reserve a stable reference version so that a complaint about changed behavior is compared against a reproducible predecessor rather than an undocumented sequence of adaptations.

Technical background

Related engineering context: Inference benchmark design; Confidence intervals.