# Differential API testing across two inference engines

Differential tests for two inference engines, comparing schemas and terminal states with an explicit differences register separate from text quality.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/provider-engine-differential-api-tests/
Language: en

## Define the shared contract

Two engines can expose a compatible endpoint while differing in parameters, defaults and extension fields. A differential test sends one fixture through two adapters and compares only properties selected beforehand. It is neither a speed competition nor evidence of identical model quality. Before execution, record the intersection of supported features, interface versions and permitted differences. If one side lacks a required mode, report an unsupported capability rather than accidentally comparing a different request. Silently removing a parameter from only one side and then declaring the responses equivalent is particularly misleading.

## Pin the envelope and limitations

Basic assertions cover HTTP status, content type, required JSON fields, identifiers and sequence termination. [OpenAPI 3.1.1](https://spec.openapis.org/oas/v3.1.1.html) helps version the schema independently of a particular client. The [vLLM online serving documentation](https://docs.vllm.ai/en/stable/serving/online_serving/) separately lists supported interfaces and limitations; a familiar endpoint name does not imply support for every field without exceptions. The second engine needs an equally versioned capability record. Save original responses alongside normalized results. Without the originals, it is impossible to tell whether a difference originated on the server or was introduced by an overly aggressive test adapter.

## Normalize only explainable differences

Random identifiers and creation times are usually unsuitable for literal equality, but their presence, type and consistency across frames remain testable. An exception register should describe the field, justification, scope and review date. An unknown difference must not automatically vanish during normalization. For example, missing usage on one side cannot become zero merely to achieve equality. Compare finish reasons using the agreed semantics first, while preserving the original values separately. An exception belongs to a specific version and scenario; it should not permanently disable an entire data category for every later release.

## Investigate generation differences

When envelopes match but prose differs, examine the model, tokenizer, chat template, sampling and execution environment. The [vLLM reproducibility guide](https://docs.vllm.ai/en/stable/usage/reproducibility/) describes repeatability limits across versions and hardware. Even an identical seed is not a universal promise of bitwise equality across arbitrary engines. For tool calling, inspect call structure and whether arguments can be parsed, rather than matching punctuation in surrounding explanations. For output limits, check the permitted terminal state. Assess semantic answer quality with a separate corpus so protocol validation does not become entangled with subjective wording preferences or random model variation.

## Preserve a decision useful artifact

The final matrix links each fixture to both engine versions, the examined property, its result and any known exception. Run a small bounded suite before migration; repeat disputed cases in the recorded environment with an explicit attempt limit. The [SGLang and vLLM comparison](/journal/sglang-vs-vllm-serving/) and [vLLM and Ollama comparison](/journal/vllm-vs-ollama-serving/) provide broader context. This matrix supports a concrete decision about moving a client scenario. It does not guarantee compatibility with every library, identical tokenization or performance under load. Those claims need separate experiments, and the exception register should remain visible after a successful migration check.