# Why prompt_tokens differs from tokenizing the visible text

Explain prompt-token differences through chat templates, control tokens and tokenizer revisions, then build a reproducible reconciliation fixture.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/provider-prompt-token-template-accounting/
Language: en

Character count does not determine prompt_tokens. Even locally tokenizing the user's text can differ from server accounting because the model receives a prepared message sequence. Roles, control markers and other chat-template elements can be present. Investigating a mismatch starts by defining what each measurement covers rather than assuming either counter is wrong.

## Describe the input before tokenization

Save an anonymized messages array with its exact role order. Do not replace it with one concatenated string, which loses conversation structure. Record tool definitions, system instructions and template options separately where they participate in preparing the selected model sequence.

The [Hugging Face guide](https://huggingface.co/docs/transformers/chat_templating) explains that chat messages become a token sequence and models use different control formats. Identical visible conversations therefore need not have identical token counts across models. Comparison requires a specific tokenizer and template revision.

## Identify the counter boundary

The [OpenRouter schema](https://openrouter.ai/docs/api_reference/overview) identifies the native tokenizer as the basis of returned token counts. This does not promise agreement with an arbitrary client-side counting library. For an owned service, define whether prompt accounting happens before templating, after templating or through a confirmed engine value.

Preserve units and provenance in the report. A field labeled approximate client estimate must not become actual consumption. If the selected path lacks a counter, record an unknown value rather than substituting zero for a convenient table.

## Build a small comparison set

Start with one user message. Add a system message, a multi-role history and one tool definition in separate steps. Change one element at a time. Record visible text, the prepared sequence where available, and the native prompt count for every variant.

The artifact is a table of fixture, revision, counting method, value and explanation. It is more useful than a screenshot of two numbers. Keep inputs short and synthetic so they can live beside the test without exposing real conversations.

## Investigate distinct branches

If differences appear only after adding a role, inspect templates and control markers. If every input changes after an update, compare tokenizer versions and special-token preparation. If only tool-bearing requests differ, investigate tool serialization at the adapter boundary.

Do not repair differences with one fixed multiplier. Message composition changes, so a correction that fits one example can fail on another. Also check whether two preparation mechanisms add the same special markers twice.

## Preserve reconciliation limits

This set explains reproducible differences between technical counters, but establishes neither request price nor correctness of every external report. Financial reconciliation needs its own rules and authoritative settlement source. Matching input counts also says nothing about answer quality.

Related topics cover [tokens and billing](/journal/router-billing-token-accounting/) and a [reproducible stack](/journal/software-stack-reproducibility/). Both depend on provenance: a model name and final number without conditions are insufficient evidence for comparison.