In this article
Valid JSON and schema conformance are different levels. Test advertised support with small positive and negative fixtures.
A model returning braces does not establish that a service honored a structured-output contract. Required properties can be missing, numbers can be strings, and extra fields can violate the expected shape. Providers should distinguish syntax validity, conformance to a particular schema and support for the selected backend mode. Validation needs an explicit outcome rather than an impression from a plausible example.
Record the advertised mode
The vLLM documentation describes structured-output settings and supported formats. Save backend, model, engine version and exact schema. Any of these can change behavior without changing the public endpoint name.
Do not equate an instruction saying “answer in JSON” with an advertised interface capability. The former evaluates model behavior; the latter also tests parameter handling and implementation constraints. Give these scenarios different fixture IDs and expected outcomes.
Test syntax separately
The first check parses the completed document with a standard JSON parser. It catches unterminated strings, surrounding prose and other formatting errors. Successful parsing does not establish that required fields exist or contain acceptable values.
For streaming, parse the correctly assembled result rather than every incomplete fragment. Otherwise normal fragmentation becomes a false structured-output failure. Also preserve terminal outcome: a truncated document must not automatically count as a complete answer.
Add a minimal schema
JSON Schema distinguishes properties, required fields and additional properties. Begin with a small object containing one mandatory number and one restricted string value. It is easier to diagnose than a large application document with many nested alternatives.
Define expected properties before running: field presence, correct type and extra-key handling under the schema. Use an independent validator. Testing only with the code that produced the answer can reproduce the same mistake on both sides and create false confirmation.
Compare positive and negative fixtures
A positive case should permit an unambiguous valid result. Negative cases examine an unsupported keyword, incompatible parameter shape or inability to satisfy the advertised constraint. Define rejection according to the service contract rather than inventing one universal status for every implementation.
The OpenRouter overview distinguishes JSON mode from schema mode in its external API. It helps name the compared capability but does not replace upstream validation. One successful simple example does not establish support for every schema variant.
Make the result a release gate
Store schema, fixture hash, validator and outcome beside adapter version. Repeat the small set after updates and compare changes explicitly. Matching shape does not establish factual correctness: a model can return a well-typed but wrong number.
The OpenRouter JSON-schema guide covers parameter use, while quantization quality checks cover substantive usefulness. Contract fixtures test something else: reproducible shape compliance and honest capability boundaries without promises about unknown models or active integration.