# Seeds and fingerprints: testing an API reproducibility claim

How to test seed and system_fingerprint in an inference API, distinguishing field support, fixed environment repeatability and limits across versions.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/provider-seed-fingerprint-contract-tests/
Language: en

## Separate three promises

The first claim is that an endpoint accepts a seed and passes it to the engine. The second is that repeated requests usually reproduce within a described environment. The third guarantees identical results under explicitly listed conditions. These promises are not interchangeable. The [OpenRouter parameter reference](https://openrouter.ai/docs/api_reference/parameters) documents public parameters and behavioral qualifications; a field's presence does not imply universal determinism across routes. Pin the exact wording of the service contract before testing. If the system does not promise repeatability across hardware or releases, an experiment should not present that promise as established merely because several answers happened to match.

## Verify actual field support

An initial test distinguishes a valid value, an omitted parameter and an intentionally invalid type. Examine the API response, server configuration and available evidence of value propagation rather than inferring support solely from prose. Short answers may match even when the seed is ignored. The [OpenRouter API overview](https://openrouter.ai/docs/api_reference/overview) describes the public envelope, but an optional field such as system_fingerprint must not be invented when a particular backend supplies no data. If a fingerprint is absent, preserve that absence. If present, record its documented meaning rather than assigning it an arbitrary interpretation as a hash of all infrastructure or proof of unchanged model weights.

## Pin the repetition scope

Record weights, tokenizer, chat template, engine release, sampling parameters, hardware and batching conditions. The [vLLM reproducibility documentation](https://docs.vllm.ai/en/stable/usage/reproducibility/) explains why release and environment boundaries matter. A short series of identical requests tests only the selected environment. Compare original tokens when available, or a precisely defined response representation; whitespace normalization can conceal genuine differences. Set repeat count and output limits before execution. Separate sequential and concurrent modes as well, because changes in work scheduling may be part of the condition under investigation rather than incidental noise that can silently be discarded.

## Change one factor at a time

The diagnostic matrix first holds the environment fixed, then changes the release or load mode separately. If the fixed series is stable while the concurrent series differs, investigate batching and scheduling. If divergence appears only after an update, inspect the relevant components and documentation rather than blaming the seed without evidence. Use the fingerprint as an additional observation: equality alone does not establish identical hidden conditions. Report the changed factor, comparison count and difference type. A small corpus cannot establish a universal repeatability probability without a sampling model and a predefined statistical evaluation method.

## Publish a bounded conclusion

A sound conclusion describes the tested scope: specific versions, execution mode, corpus and observed agreement. List unsupported promises and fields the server does not expose separately. See [software stack reproducibility](/journal/software-stack-reproducibility/) and [model update canaries](/journal/model-update-canary/). A protocol seed test does not replace quality evaluation, tool call safety checks or API schema migration. Changing a guarantee's scope requires updating the contract and fixtures rather than merely increasing repetition count. This approach gives users verifiable information without turning best effort behavior into an unsupported service commitment.