Reliability, Cancellation and Recovery
OOM, worker loss, malformed inputs, partial streams.
Chapter 29 split phases. This chapter handles failures.
Failure modes
Inference services face OOM, worker loss, malformed inputs, and partial streams. Each requires explicit handling.
Cancellation
Cancellation stops work and releases resources. It must propagate to all stages. gRPC cancellation shows that signalling alone is insufficient; applications must propagate it and release their own resources.
Timeouts
Timeouts prevent unbounded work. They should be enforced at request and operation levels.
Retries
Bounded retries can recover from transient failures. They must not mask persistent errors or cause duplicate side effects. Retryable transport failure differs from unsafe replay of a non-idempotent request.
Recovery
Recovery restores service after failure. It includes restarting workers, reloading models, and clearing state. Ownership of model replicas, KV state, request state, and control-plane metadata must be explicit.
Research context
The selected sources distinguish cooperative cancellation from automatic recovery. gRPC cancellation requires application-level propagation. Ray Serve documents replica, actor, node, and control-plane recovery, but those guarantees are runtime-specific. DeepInfer’s deployment note describes production failure handling and disaggregated serving, while noting that some production components are not public.
The practical design rule is to define cancellation as cooperative propagation through queue, scheduler, kernels, cache, and downstream calls, and to separate retryable transport failure from unsafe replay.
Experiment
The repository example simulates success, retry, cancellation, and timeout. Resources were released in all cases. Tests verify behaviours and cleanup.
Limitations
- Simulation only.
- No real fault injection.
- No distributed recovery.
- No OOM handling.
- Reported production behaviour from vendors is not independently reproduced.
Key terms
- Cancellation: stopping work.
- Timeout: time limit.
- Retry: repeated attempt.
- Recovery: restoring service.
- Resource cleanup: releasing state.
Reader exercises
- Add OOM simulation.
- Test partial stream cleanup.
- Implement exponential backoff.
- Explain why retries can worsen overload.
References
- gRPC cancellation guide - official protocol documentation.
- Ray Serve fault tolerance - official runtime documentation.
- DeepInfer inference deployment Q&A at 774ce244 - pinned first-party technical note.
Implementation status
- Manuscript:
content/books/inference/30-chapter.md - Runnable example:
examples/ch30/ - Tests:
tests/test_ch30_reliability.py - Evidence:
research/ch30/sources.mdandresearch/ch30/results.json - Verification command:
python -m unittest discover -s tests -v
The chapter is a draft. It does not claim production reliability. The next chapter will examine protocols.