← Inference From First Principles

Designing Meaningful Inference Benchmarks

Workload mixes, baselines, uncertainty.

Chapter 39 addressed reproducibility. This chapter designs benchmarks.

Realistic workloads

Benchmarks must reflect real usage: prompt lengths, output lengths, concurrency, and task types.

Baselines

A benchmark needs a baseline. Compare against a simple, well-understood implementation.

Statistical uncertainty

Repeated trials and confidence intervals quantify uncertainty. Single runs are insufficient.

Research context

Freeze the system under test, model, revision, quality target, workload, arrival process, cache state, and measurement points. Report TTFT, TPOT, ITL, end-to-end percentiles, throughput, goodput, errors, and power where relevant. MLPerf separates scenarios, quality targets, latency constraints, throughput, system metadata, and power. vLLM documents cache-related benchmark hazards. Separate repository-reported reference graphs from measurements reproduced by this book.

Experiment

The benchmark suite ran 6 trials over 2 workloads. Mean latency 0.32, CI [0.232, 0.408]. Tests verify trials, CI, report, and serialisation.

Limitations

  • Synthetic workload.
  • No real model.
  • No paired comparison.
  • No hardware.

Key terms

  • Workload: set of requests.
  • Baseline: reference implementation.
  • Confidence interval: uncertainty range.
  • Trial: repeated run.
  • Paired comparison: same inputs across systems.

Reader exercises

  1. Add a real workload manifest.
  2. Run paired comparisons.
  3. Explain why synthetic benchmarks mislead.
  4. Design a benchmark report.

References

Implementation status

  • Manuscript: content/books/inference/40-chapter.md
  • Runnable example: examples/ch40/
  • Tests: tests/test_ch40_benchmark.py
  • Evidence: research/ch40/sources.md and research/ch40/results.json
  • Verification command: python -m unittest discover -s tests -v

The chapter is a draft. It does not claim benchmark validity. The next chapter begins Part 8 capstone.