Validate JSON Schema to GBNF conversion in llama.cpp

Converting JSON Schema to GBNF for llama.cpp does not establish that every schema constraint is enforced. The converter supports a subset, and some constructs can be skipped. Acceptance needs valid and invalid fixtures, an independent validator, and the generated grammar. Check constraint meaning separately from successful conversion or a model’s persuasive textual explanation of the returned object.

Define grammar guarantees

GBNF restricts output sequences according to its rules. The grammar guide describes schema conversion and limitations. It is not a general business-rule engine or a promise to support every JSON Schema version. A rule involving database existence or external state still requires application validation. Even well-formed JSON with allowed fields may be unsuitable for the task. Separate syntax, schema, and domain requirements before testing so that success at one layer cannot silently stand in for the others.

Inventory schema keywords

Keep the original schema and the constructs actually used. Consult documented unsupported features and converter warnings. No exception does not prove complete support: the guide specifically discusses constraints that may be skipped. Repeat keyword checks after version changes. Avoid rewriting prompt and schema together merely to pass a test, because that obscures which layer enforces the rule and whether it will survive a different user input. Preserve conversion output as an artifact rather than regenerating it invisibly during every investigation.

Build a negative-case matrix

Use an object with a required field, enumeration, and nested structure. Add a valid example, a missing field, an invalid enumeration value, and a wrong type. Include an invalid object violating only the complex constraint that matters to the application. This defines testing without claiming support for a particular keyword. Check fixtures with an independent schema validator and grammar validation. If results disagree, identify the exact rule instead of declaring either all constrained decoding broken or every original constraint guaranteed.

Check the server contract

The server README documents grammar and schema controls. Verify the request field and endpoint for the installed build. A schema does not necessarily supply a semantic task description to the model, so the prompt must still explain the desired object. During streaming, intermediate fragments need not be complete JSON. Save the assembled answer and termination condition. Output truncated by its budget cannot establish a grammar-conversion failure until the premature ending has been investigated independently.

Assign application responsibilities

If the conversion cannot express a required constraint, validate it explicitly after generation and define rejection handling. An invisible unconstrained retry is not an equivalent contract. Keep schema, generated grammar, versions, warnings, and negative-fixture results with the release. Compare these artifacts after engine updates. The result should identify rules enforced by grammar, an independent validator, and domain logic. This makes constrained generation predictable without claiming that every schema-endpoint response necessarily satisfies all of the author’s intentions or every downstream requirement under arbitrary prompt changes.

Structured output through OpenRouter: when to use a JSON schema; llama.cpp server: a local API from a GGUF model.