# High-volume Haiku traffic needs failure slices, not one average

Why high-volume Haiku 5.5 evaluation needs failure slices, consequence-aware routing and explicit uncertainty beyond a pooled average.

Author: Retrivium editorial team
Language: en
Analysis published: 2026-11-03
Event date: 2026-10-07
Original announcement: https://www.anthropic.com/claude-haiku-5-5
Canonical URL: https://retrivium.ai/retrospectives/haiku55-volume-evaluation-slices/

## Scale made the average less informative

[Opus 5.5's September 22, 2026 release](https://www.anthropic.com/claude-opus-5-5) and [Sonnet 5.5's September 28 release](https://www.anthropic.com/claude-sonnet-5-5) preceded [Haiku 5.5 on October 7](https://www.anthropic.com/claude-haiku-5-5). The smaller model's high-volume positioning makes evaluation composition especially important. A good aggregate result can coexist with a weak subset that matters operationally. When a system handles repetitive work, the question is not just whether it performs well on the most common case. It is whether consequential exceptions remain visible in the evidence used to accept the workflow.

## Frequency and consequence are separate dimensions

An illustrative classification workload contains ordinary requests and a less common category requiring human review. The common category can dominate a pooled accuracy figure. A model that handles it well may still route the review-required cases incorrectly. No real frequency or failure rate is assumed here. The point is structural: weighting by occurrence and weighting by consequence answer different questions. Both can be useful, but neither should silently replace the other when deciding whether an automated route is suitable for a particular operation.

## Slices should follow plausible failure mechanisms

A useful evaluation slice is not an arbitrary demographic or formatting label attached after seeing the outcome. It corresponds to a reason the task may behave differently, such as missing context, conflicting identifiers or an exception to the normal rule. These distinctions help explain a failure and suggest a bounded correction. An excessive collection of tiny slices can also produce noisy conclusions. The evaluator should retain enough examples and uncertainty information to avoid turning a single unusual result into a broad claim about the model.

## A routing fixture with explicit abstention

A practical illustrative fixture includes clear cases, ambiguous cases and cases whose required information is absent. The acceptance contract permits an unresolved or review-needed outcome where appropriate. Inspect both incorrect confident decisions and unnecessary escalation of easy work. This prevents a candidate from appearing reliable merely by declining everything, or efficient merely by choosing an answer for every input. The fixture supplies no benchmark score. It defines observations that connect model behavior to the intended routing policy and the cost of handling its exceptions.

## Monitor the workload that actually arrives

The October announcement provides a reason to investigate economical repeated inference, but release positioning is not evidence about an application's input distribution. After any local acceptance, the mix of tasks can change. Retaining slice definitions and the reasons behind them makes later changes interpretable. The retrospective lesson is to connect scale with observability of failure, not just with a larger aggregate sample. A high-volume model choice becomes defensible when common-case efficiency and consequential-case handling are both supported by evidence rather than hidden inside one average.

## Related reading

- [Designing task slices](/journal/inference-benchmark-method/)
- [Avoiding overconfident small-sample claims](/journal/gpu-benchmark-confidence-interval/)