← Inference From First Principles

Adding Multiple Model Backends

Backend capabilities and contracts.

Chapter 42 integrated scheduling. This chapter adds multiple backends.

Backend contract

A common contract allows swappable backends. The PyTorch toy backend passes conformance. vLLM and SGLang are stubs with explicit skip reasons.

Capabilities

Each backend reports capabilities. Unsupported features raise errors rather than silently falling back.

Research context

Define a backend contract for tokens, logits/sampling ownership, cache semantics, streaming events, errors, and capabilities. Use conformance tests to expose protocol and tokenizer differences. Fail closed on unsupported model/hardware/feature combinations. Transformers documentation and gpt-oss verification provide external conformance references; DeepInfer provides a pinned first-party pattern but is not independently measured.

Conformance

Conformance tests verify that a backend implements the contract. The example ran 3 backends: PyTorch passed, vLLM and SGLang unavailable.

Experiment

Tests verify conformance, stub behavior, demo, and serialisation.

Limitations

  • Stubs only for vLLM/SGLang.
  • No real backends.
  • No performance comparison.
  • No conformance for real providers.

Key terms

  • Backend: model implementation.
  • Conformance: contract compliance.
  • Stub: placeholder.
  • Capability: supported feature.
  • Skip reason: explicit unavailability.

Reader exercises

  1. Implement a vLLM adapter.
  2. Add capability reporting.
  3. Explain skip reasons.
  4. Design a conformance suite.

References

Implementation status

  • Manuscript: content/books/inference/43-chapter.md
  • Runnable example: examples/ch43/
  • Tests: tests/test_ch43_backends.py
  • Evidence: research/ch43/sources.md and research/ch43/results.json
  • Verification command: python -m unittest discover -s tests -v

The chapter is a draft. It does not claim real backend integration. The next chapter will benchmark against established engines.