Implementing Scheduling and Caching

Chapter 41 designed the runtime. This chapter integrates scheduling and caching.

Integration

The runtime combines admission, scheduling, caching, and cancellation. Feature flags control which components are active.

Deterministic scenarios

Deterministic concurrency scenarios allow reproducible testing. The example schedules 4 of 5 requests and cleans up cancellation.

Research context

Co-design scheduling and cache allocation; neither can be treated as an isolated queue or allocator. Make admission, preemption, prefix sharing, reclamation, and fairness observable state transitions. Prove semantic parity and isolation before claiming performance improvements. Orca introduced iteration-level scheduling; PagedAttention and SGLang extend cache and scheduling integration.

Experiment

Tests verify scheduler limits, cache, integration, and serialisation.

Limitations

• Toy integration.
• No real concurrency.
• No performance.
• No production cleanup.

Key terms

• Integration: combining components.
• Feature flag: toggle.
• Scheduler: request ordering.
• Cache: stored state.
• Cleanup: resource release.

Reader exercises

1. Add a feature flag for caching.
2. Test cancellation cleanup.
3. Explain integration risks.
4. Design a deterministic scenario.

References

• Orca: A Distributed Serving System for Transformer-Based Generative Models - peer-reviewed systems paper.
• Efficient Memory Management for Large Language Model Serving with PagedAttention - peer-reviewed systems paper.
• SGLang: Efficient Execution of Structured Language Model Programs - peer-reviewed systems paper.

Implementation status

• Manuscript: content/books/inference/42-chapter.md
• Runnable example: examples/ch42/
• Tests: tests/test_ch42_integration.py
• Evidence: research/ch42/sources.md and research/ch42/results.json
• Verification command: python -m unittest discover -s tests -v

The chapter is a draft. It does not claim production integration. The next chapter will add multiple backends.