Structured Outputs and Constrained Decoding
Schemas, token constraints, validation, repair.
Chapter 34 handled tool calls. This chapter ensures outputs conform to schemas.
Schemas
Structured outputs require a schema. The runtime must validate generated JSON against it.
Token-level constraints
Constrained decoding restricts tokens to those that keep the output valid. This can guarantee syntactic validity but not semantic correctness.
Validation and repair
Validation checks output after generation. Repair attempts to fix invalid output. Validation is safer; repair can introduce errors.
Research context
The selected sources distinguish grammar compilation, token masking, parser state, and schema validation. XGrammar and guided-generation work show that syntactic conformance can be efficient, but JSONSchemaBench shows that schema-valid JSON can still be semantically wrong or unsafe.
The design rule is to separate syntactic validity from semantic, factual, and business-rule correctness, and to benchmark coverage and compile overhead as well as tokens per second.
Experiment
The repository example validates JSON against a schema and filters tokens. Valid output passed; missing field and invalid JSON failed. Tests verify validity, invalid JSON, FSM filter, and serialisation.
Limitations
- Toy schema.
- No real constrained decoding.
- No semantic validation.
- No repair.
Key terms
- Schema: output structure.
- Constrained decoding: token restriction.
- Validation: post-generation check.
- Repair: fixing invalid output.
- Syntactic validity: format correctness.
Reader exercises
- Add nested schema validation.
- Implement a finite-state machine for JSON.
- Explain why syntactic validity is not semantic correctness.
- Compare constrained decoding with validation.
References
- Efficient Guided Generation for Large Language Models - research paper.
- XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models - research paper.
- JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models - research paper.
Implementation status
- Manuscript:
content/books/inference/35-chapter.md - Runnable example:
examples/ch35/ - Tests:
tests/test_ch35_structured.py - Evidence:
research/ch35/sources.mdandresearch/ch35/results.json - Verification command:
python -m unittest discover -s tests -v
The chapter is a draft. It does not claim production structured output. The next chapter will examine metrics.