WER and CER for speech recognition: normalize text without hiding errors

WER and CER compare transcripts only under a declared normalization policy, especially for Russian text containing numbers, punctuation and spelling variants. A lower normalized error rate can reflect a changed scoring rule rather than better recognition of the information users need from the recording.

Define the counted units

The Hugging Face audio evaluation chapter explains word and character error rates. WER counts edits over word units; CER uses characters. Neither directly measures whether a technical instruction remains correct. A mistaken product digit can matter more than several harmless formatting differences. Preserve examples alongside aggregate scores so unit-level counting does not hide practical consequences or obscure the task being evaluated.

Declare normalization before scoring

The JiWER implementation and documentation expose transformation pipelines. Decide how to handle case, punctuation, spaces and numeric forms consistently in reference and hypothesis. For Russian, explicitly consider whether the letters named yo and ye are equivalent for the task. A policy appropriate for general speech may be inappropriate when exact orthography or identifiers form part of the required output.

Use an informative Russian fixture

Prepare reviewed references with a spoken amount, a product code and a sentence containing a negation. Compare a numeric rendering with a word-form rendering, then inspect a genuinely wrong digit separately. The scoring policy may normalize representational differences, but must not erase the wrong identifier or negation merely to reduce errors. Keep raw transcripts so every transformation remains auditable and meaningful differences stay available for inspection.

Difference	Evaluation decision
Punctuation only	Normalize if outside the task
Number words versus digits	Apply a declared equivalent-form policy
Wrong product character	Preserve as a substantive error
Missing negation	Inspect semantic impact separately

Aggregate without hiding the workload

Report how utterance results are combined and retain slices for identifiers, quiet speech and language mixtures. A corpus average can improve while critical short instructions regress. Empty references, repeated words and segmentation boundaries need defined treatment. Use the chosen library’s behavior explicitly instead of reproducing a metric name with an undocumented calculation. Different preprocessing can make apparently comparable scores describe different evaluation tasks.

Publish text and scoring changes

Save reference revision, normalization steps, metric implementation and representative alignments. Compare raw and normalized error analyses where useful, explaining why differences are ignored or retained. Keep semantic checks for instructions outside the arithmetic metric. This makes WER and CER reproducible diagnostic tools without presenting them as universal measures of transcript usefulness. A permissive normalization rule must not conceal recognition mistakes that change a product, a condition or the meaning of the recorded instruction.