# The logprobs contract: aligning tokens and returned scores

Check logprobs and top_logprobs in an inference API across opt-in behavior, output-position alignment, streaming and limits on interpreting scores.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/provider-logprobs-response-consistency/
Language: en

Logprobs adds probability information to text, but a field's presence alone does not establish correct implementation. A server may accept the option without returning data, or return a shape clients associate with the wrong positions. Provider checks should establish response structure and consistency on a small set without treating token probabilities as a universal model-quality score.

## Record parameter dependencies

The [OpenRouter documentation](https://openrouter.ai/docs/api_reference/parameters) describes logprobs as opt-in and makes top_logprobs dependent on enabling it. Preserve that relationship in a fixture. A request without the capability, one enabling it and an incompatible combination are distinct cases rather than random versions of one prompt.

Do not transfer one external interface's limits to every upstream. Record model, engine and adapter version in your matrix. Unsupported capabilities need the classification defined by the contract. Missing promised data after request acceptance warrants investigation rather than an automatic success result.

## Choose a simple output

Begin with a short synthetic generation and a small output limit. Save text, returned positions and available token details. Avoid complex tool calls or long documents initially, where association errors are harder to separate from other transformations.

Define a position in the report: engine token, text fragment or another documented unit. Spaces in the answer cannot establish it. A displayed character or client string need not correspond to one token position.

## Compare enabled and absent modes

The no-logprobs variant establishes the baseline envelope. The enabled variant checks for the promised structure. A separate negative fixture supplies the wrong parameter type or a combination forbidden by the contract. These validate API handling rather than deterministic generation.

The [vLLM interface overview](https://docs.vllm.ai/en/stable/serving/online_serving/) illustrates concrete compatibility limits. One server result does not establish support across a model family. After changing a backend, repeat the checks and record exceptions even if ordinary text remains successful.

## Account for streaming assembly

In streaming mode, score information may accompany output fragments. Assemble events under the selected interface before checking alignment. A network-read boundary does not establish a token position. Reordered or duplicated events can corrupt associations while leaving plausible-looking text.

[RFC 8259](https://www.rfc-editor.org/rfc/rfc8259.html) defines JSON numbers and objects. Successful parsing validates syntax rather than the meaning of returned probabilities. Preserve original envelopes to distinguish server issues from client-library restructuring.

## Limit interpretation

Even correct scores relate to a model, context and particular distribution. They establish neither factual truth nor task quality. Do not compare implementations without checking definitions, units and transformations.

The [Python request example](/journal/openrouter-python-request/) supplies request context, while [quality evaluation](/journal/quantization-quality/) addresses usefulness. A conformance artifact records parameters, data shape and discovered boundaries, keeping observable interface behavior separate from the substantive reliability of an answer.