From Inference Loop to API

Twenty-nine awkward requests were sent to a first-draft inference service, the kind that takes the framework’s defaults: a body that is not JSON, a prompt that is empty, an option spelt wrongly, a temperature of NaN, a prompt of 4,000 emoji. It answered 200 to eight of them. Four of those answers were not what the caller asked for: "5", 5.0 and true were all accepted as a token limit, and a request for max_tokens: 2 (a misspelling of max_new_tokens) returned sixteen tokens, the default of the field that did exist. It answered 500 to four others, and none of its twenty-one errors came in a shape the service had chosen or documented: most were the framework’s detail lists, the rest one-line strings.

A second service, built from the same engine, was sent the same twenty-nine. It answered 200 to four, all of them requests that were acceptable as asked (one carried a warning). It answered 25 with an error, all in one documented shape, with a request id, the name of the parameter at fault and a type a client can switch on. It answered 500 to none.

The engine behind both services was identical. This chapter is about the difference: what has to be decided at the boundary between a loop that generates tokens and a program that calls it over a socket, and what each decision costs or saves when it is measured.

The question

Chapter 6 gave the engine a Request, a Result and a rule for what a backend is. Chapter 24 gave it an admission function that refuses what the runtime cannot do. Neither says what the engine does when the caller is not another function in the same process, but a client on a network that can send anything at all.

Five questions follow. Where does a request’s time go once it crosses a socket, and is the interface a significant part of it? Can a limit written in characters protect a budget written in tokens? What should a client see when the request is wrong, and does the framework’s default say it? How may the response change next month without breaking the clients that read it today? And does what comes back over the wire match what the engine computed?

Where this sits

The Inference Lab reaches Part 5 here. Chapter 6 supplied the engine, Chapter 10 the cache behind its backend contract, Chapter 24 the admission rules and the lesson that a silently ignored option is a fallback. This chapter wraps them in a service and keeps the Chapter 6 Result fields (usage, timing, provenance) as the basis of the response.

It also pays a debt. Chapter 9 measured time to first token from the moment the prefill call began, and noted that a real figure includes tokenisation, queueing and the network, to be added “when it has a service to measure at”. This chapter can measure the non-streaming part of that: the stages between arrival and the first token, for a request that asks for exactly one token. Streaming, and so true time to first token for a long answer, belongs to Chapter 27. Scheduling and what to do when requests arrive faster than the engine can serve them belongs to Chapter 26; the service here has one worker and a lock, and says so.

The service is deliberately small: one model (GPT-2 small, behind the Chapter 10 cached backend), one process, no authentication. It exists so that the decisions can be measured rather than argued.

A request’s life

A request enters, is read, parsed, validated, checked against the budget, waits for the engine, runs, is serialised and leaves. The service stamps the arrival time and a request id on the way in, and records the time between each pair of stages. A client that asks for one token sees the whole path with almost nothing else on it.

I measured it over loopback HTTP from a separate process, with a persistent connection, 20 requests per row after three warm-ups, on GPT-2 small on the RTX 2060. The table gives medians in milliseconds.

Three readings. First, the interface is cheap. Parsing, validating, tokenising and building the response come to about 0.7 ms and the transport to about 1.2 ms: under 2 ms in every row, about 9% of the one-token request and 0.08% of the 128-token one. Second, the engine’s cost is the rest: 91% of the one-token request and 99.9% of the 128-token one. The prefill of this short prompt costs about the same as one further decode step, and each additional token costs about 19 ms, so a request is, to a first approximation, 19 ms times the number of tokens it asks for. Third, and the reason to keep the stage times in the response, nothing in the interface explains the variation in a client’s experience: it comes from the length of the answer, which the caller chose. (The numbers vary from run to run on this shared card: an earlier clean run of the same experiment had a 128-token median of 2.89 s against 2.46 s here. The ratios between rows are stable; the absolute figures are not.)

The one-token row is the nearest non-streaming stand-in for time to first token: 22.7 ms at the client against 19.0 ms for the prefill inside the engine. The difference is parse, tokenise, selection, response building and transport; none of it is large, which is the finding.

Connecting is not free, and the name matters

The same client is slow on the first connection in a way that has nothing to do with the model. A raw TCP connect to 127.0.0.1 took a median of 0.64 ms. A connect to localhost took 2,036 ms, and a request on a fresh connection to localhost took 2,047 ms against 1.61 ms for the same request to 127.0.0.1. On a kept-alive connection the two names cost the same (1.15 and 1.36 ms).

The measured pieces are these: on this machine the name localhost resolves to ::1 first and 127.0.0.1 second; the server listens on IPv4 only; and a connect to a port with nothing listening takes about 2.0 s on either address (I measured both, 2,030 and 2,021 ms). What I did not do is trace the connection, so the explanation (the IPv6 attempt is refused slowly and the client then falls back) is the reading that fits the numbers, not a captured sequence. The practical conclusion does not depend on it: a health check, a test or a first request written against localhost can be two seconds slower than the identical one against 127.0.0.1, and the commands in this chapter use the address.

The event loop and the health check

A service also answers requests that are not generations, and a load balancer will ask whether it is alive. Where the engine runs decides what that check sees. FastAPI runs a plain def route in a worker thread; an async def route runs on the event loop, and a blocking call inside it blocks everything on that loop. In lab mode the service exposes both, with the same body, so they can be compared. A client polled /health every 20 ms during a 256-token generation, three times for each route.

On the async def route one health check waited for the entire generation, 4.9 to 5.6 seconds, and the median was still two milliseconds because there were so few of them. A liveness probe with a timeout shorter than the generation would fail during a long answer, and an orchestrator that acts on a failed probe would restart a healthy process. The service therefore keeps /health (the process is up) separate from /ready (the model is loaded, with a 503 and a Retry-After while it is not) and runs the engine in a thread. That is a mitigation and not a design for concurrency: the engine is still one at a time, which the next sections come back to.

The contract

Requests

The request schema is the first place where the loop’s assumptions meet the world. The Chapter 6 Request.__post_init__ checked that values were in range; the API has to say what a type is. The service’s schema is a pydantic model with strict JSON types, bounded fields and unknown fields rejected:

class GenerateRequest(BaseModel):
    model_config = ConfigDict(extra="forbid", strict=True)
    prompt: str = Field(min_length=1, max_length=4000)
    max_new_tokens: int = Field(16, ge=1, le=256)
    temperature: float = Field(0.0, ge=0.0, le=2.0)   # 0 means greedy
    top_k: Optional[int] = Field(None, ge=1, le=1000)
    top_p: Optional[float] = Field(None, gt=0.0, le=1.0)
    seed: int = Field(0, ge=0, lt=2**31)
    stop: List[Annotated[str, StringConstraints(min_length=1, max_length=32)]] = Field(default_factory=list, max_length=4)

Two of those lines are decisions that the framework’s defaults would have made the other way. The first-draft variant, the same fields with strict=False and extra="ignore", accepted "5", 5.0 and true as a token count and silently ignored max_tokens. That last case is the API-level twin of Chapter 24’s temperature with do_sample=False: a parameter the library did not understand and did not mention. A request that names an option the service has not got should be refused with the option’s name. The strict model does so: a 422 with param: "max_tokens".

Strictness has edges. The strict float accepted the JSON integer 1 for temperature, as it should; a whitespace-only prompt and a prompt containing a NUL character are both admitted by both services. Those are policies, not accidents: the tokeniser handles them and so I let them through, but a service whose users send form input may well want to refuse them. The point is that they are now a choice written in one place.

temperature: 0 is defined as greedy decoding (Chapter 3’s explicit form of temperature zero) and then top_k, top_p and a non-zero seed have no effect. The service does not reject that combination, because it is harmless and callers send defaults. It does not ignore it either: the response carries a warnings entry, top_k has no effect when temperature is 0 (greedy decoding). Chapter 24’s rule was that a request is refused, warned about or silently altered, and the third is the one to avoid.

Responses

A generation response has to say more than the text. The service returns, in this order of importance: an id equal to the X-Request-Id header; the model identity (name, snapshot and the SHA-256 digest of the backend’s manifest, the Chapter 2 idea as Chapter 6 applied it, so that a result can be traced to an artefact and not a name); the text and the token_ids that produced it; a finish_reason (length, stop or eos); usage with prompt, completion and total tokens; warnings; and, since version 1.1, timing with the stage times above. There is also a Server-Timing header with the total, in the syntax of the W3C’s Server Timing draft (a Working Draft, not yet a Recommendation).

The token ids are in the response for a reason the fidelity measurement below makes concrete. Returning only text would throw away the one representation the engine actually produced.

Changing the contract

The starter chapter said that adding fields is safer than changing or removing them. That is true, with a condition, and the condition is who is reading. I wrote a small classifier for the claim and tested it against behaviour.

The rule is containment. A request change is compatible if the server still accepts everything the old schema accepted, and means the same by it: so raising a limit is fine, lowering it is not; adding an optional field is fine, adding a required one is not; changing a default is not, because the same request now means something else. A response change is compatible if everything the server may now send still fits what an old client reads: adding a value to an enumeration is not, because a client that switches on finish_reason meets a value it has no branch for; removing a value is. diff_schemas applies those rules to two JSON Schemas, as pydantic produces them.

To check that the rules say what clients experience, I made 19 changes to a miniature of the real schema (11 to the request, 8 to the response), and for each one sent the old client’s requests to the new model, or the new model’s output to the old client. The classifier and the behaviour agreed on all 19.

The bold cell is the condition. Adding a field is compatible with a client that ignores what it does not know, and breaks one that was written to reject it. Neither is wrong; both exist. A published contract has to say which kind of reader it assumes, and the service’s version 1.1 depends on the first. The real version 1 to version 1.1 change (one field, timing) classifies as compatible, and it is: the tolerant client reads it.

The Anthropic API documentation takes the same position in a different place. It says of its error responses that the values of the type field “may expand” over time, which asks clients to tolerate a type they have not seen. That is documentation of an interface, not evidence about how its service is built, but it is the same rule: readers ignore what they do not recognise, writers only add.

Budgets

A request that cannot succeed should fail before it costs anything. For generation the cost is set by tokens, and the schema’s limit is written in characters. I measured how far apart they are with GPT-2’s tokeniser, on 1,000-character samples.

The schema’s cap is 4,000 characters, which seems generous against a window of 1,024 tokens. 4,000 characters of ordinary prose (the first sample repeated to length) is 972 tokens: it fits, barely. The same cap admits 1,532 tokens of source code, 4,000 of spaces, 8,000 of Cyrillic or emoji and 12,000 of CJK, and GPT-2 has 1,024 positions. A character cap that is safe for the worst of these (three tokens per character) with 256 tokens reserved for the answer is 256 characters, which would refuse a single ordinary paragraph of 1,000 characters, about 230 tokens. No character limit protects a token budget and also admits real prompts.

So the service counts. After parsing, it tokenises the prompt (0.2 ms, in the table above) and applies one function, check_budget, which returns the parameter at fault and a message with the numbers: 901 prompt tokens plus 256 new tokens is 1157; the context window is 1024. The prompt limit (1,000 tokens in the lab’s default) bounds the cost of a prefill; the total bounds the positions. The two are separate rules because they protect different things, and the lab’s Limits keep them separate. (The first version of the defaults made the second rule unreachable: a prompt limit of 768 plus the 256 new-token maximum is exactly 1,024, so no request could break the total without first breaking the prompt limit. The probe that was meant to exercise the total rule passed through. It is the kind of mistake a table of probes is for.)

What the budget saves, when it is missing, was measured on the CPU with a request that has a 901-token prompt and asks for 256 more. The first-draft service ran the generation until position 1,024 and then raised an IndexError, which became a 500, after 12.06 seconds. The hardened service refused the same request in 9.3 ms. Those are single runs on the CPU and the ratio, not the figures, is the point: the unchecked request spends the whole cost of the work it cannot finish. I did not run this case on the GPU, where an out-of-range index is reported differently, so the figures are for the CPU only.

Where the limit sits is a decision. Here the window is typed into Limits by hand for GPT-2. A service for several models would read it from the model’s fingerprint (Chapter 24) and refuse a request whose prompt, plus its reserved output, does not fit the model the request names.

Errors

What a client can act on

A client has three questions of a failure: whether to retry, whether to change something, and whom to tell. The status code answers the first, a stable type the second, and a request id the third. The service uses one envelope for everything it can say:

{"error": {"type": "context_length_exceeded", "message": "901 prompt tokens plus 256 new tokens is 1157; the context window is 1024", "param": "max_new_tokens", "request_id": "gen-..."}}

and the statuses follow one rule about what the server could do with the request: 400 when it could not parse it (malformed_json); 413 when it declined to read it (payload_too_large); 415 when the content type is not JSON; 422 when it parsed and understood the request and could not accept it (invalid_request, unsupported_input, context_length_exceeded); 405 and 404 for the method and the path; 503 with Retry-After while the model is loading; and 500, always internal_error, for what the service did not foresee, with the detail written to the server’s log under the same request id and nothing of it in the body.

Those meanings are from RFC 9110: 413 is a body over the server’s limits, 415 a format it does not support for that resource, 422 a request whose type and syntax it understands but whose instructions it cannot follow, 503 a temporary inability with an optional Retry-After. That 422 is the status FastAPI uses for validation failures is a convention of the framework; the RFC does not mandate it, and the service could have used 400 provided it did so consistently. What matters is that the rule is written down and every failure follows it. The Anthropic documentation makes a different choice of the same kind (400 invalid_request_error for most request faults, a top-level request_id, 413 request_too_large at 32 MB for its Messages endpoint); neither is the standard way, and a client reading either one needs the documented rule.

What the defaults did

The twenty-nine probes, a selection:

Totals, over 29 requests each (the valid request excluded): the first-draft service answered 200 eight times, 422 fourteen, 500 four, 404 once and 405 twice, and none of its 21 errors was in a common envelope. The hardened service answered 200 four times, 422 twenty, 400 once, 404 once, 405 twice and 415 once; all 25 of its errors were in the envelope; neither service put a traceback or a file path in any body.

Two of the first-draft 500s deserve a closer look because the cause is not what one would guess. For NaN and for a lone surrogate, the framework correctly noticed the invalid value and built a 422 response. That response, by default, includes the offending input. NaN cannot be written as JSON, and a lone surrogate cannot be encoded as UTF-8, so building the error failed and the client saw a 500. The first-draft service turned a client mistake into a server fault by echoing it. The FastAPI documentation recommends not returning the raw validation error because it can carry details about the system; what I measured is a second reason, and also a third: the echo is as large as the request.

Large bodies

A field limit of 4,000 characters is checked after the body has been read and parsed. The size of a body is therefore not limited by it. Against a real server on the CPU, a body of 100 MB was read in full by the first-draft service, which then answered 422 after 1.4 to 1.5 seconds with a 104,857,751-byte response, the body returned to its sender, and the server’s process had grown by about 400 MB. At 20 MB it took 0.51 to 0.55 s, grew 76 to 82 MB and returned 20,971,671 bytes; at 1 MB, 0.29 s and 3 to 5 MB. Declared or chunked made no difference.

The hardened service has a small pure-ASGI middleware, BodyLimit, which rejects on Content-Length when it is declared and counts bytes when it is not (a chunked body declares nothing). With a limit of 64 KiB, a 100 KB body got a 149-to-152-byte 413 under both framings, and the server’s memory did not measurably move. For 1 MB, 20 MB and 100 MB bodies the client did not get a status at all: the httpx client raised a ReadError, a connection reset, while the server’s memory again did not move.

My reading is that the server answers and closes the connection while the client is still writing, and the client sees the reset before it reads the answer; RFC 9110 says a server may close the connection when it declines an oversized request, which allows that. I did not capture the traffic to confirm it, and a different client library may behave differently. The finding stands without the mechanism: an early rejection protects the server and does not guarantee that a client with a large body learns why. The limit that matters most in practice belongs in front of the service (a proxy, in the layers I did not build), and the service’s own limit is the backstop.

What crosses the boundary

Is the answer the engine’s answer?

“Success at one stage does not imply the next” applies to the wire. I sent 40 paragraphs of this book through the service as 32-token requests, greedy and sampled (temperature 1.0 with a different seed for each), and ran the same requests directly through the same engine in another process.

The token ids matched in 40 of 40 greedy requests and 40 of 40 sampled ones, and so did the text. The reported usage equalled an independent count of the prompt and completion tokens in every request, and the total equalled their sum. A seed means the same thing in two processes (Chapter 6’s per-step derivation hashes the pair, so the stream does not depend on the process). The interface changed nothing.

It did change one thing a client might expect. The returned text, tokenised again, was not always the sequence of ids the model produced: 39 of 40 greedy responses round-tripped, and 33 of 40 sampled ones. The same counts held when the prompt and the text were joined and tokenised together, which is what a client does when it feeds an answer back as the next prompt. I did not inspect the seven failures, so I can only offer the standard reading as a hypothesis: the model can emit a sequence of tokens that is not the tokeniser’s canonical encoding of the text the tokens spell, and sampling at temperature 1.0 does it more often than the argmax does. The consequence is established by the measurement either way. A client that sends back text instead of ids is sending the model something it did not produce, and a service that returns only text has discarded the record that would have shown it. That is why token_ids is in the response.

One engine, many callers

The service has one worker, a threadpool for its routes, and one cached backend. The Chapter 10 cache keeps its state behind the stateless contract next_logits(ids): it checks that the cache holds exactly the prefix it was given, and reuses it if so. That is safe for one caller. I put four threads on one backend, each running a different 24-token greedy request, 12 rounds of four (48 requests), and compared each result with its solo run.

Without a lock, all 48 requests failed in each of three runs. In the three runs 4, 13 and 15 of the 48 produced a wrong answer with no error at all; the others raised a RuntimeError. With a lock around the engine call, none of 48 was wrong or failed, in each run. (An earlier run that overlapped another job on the machine gave 18 wrong and 30 exceptions; the pattern was the same.)

This is the argument for the lock in the service, and the measurement of what its absence costs, and nothing more. It does not test the service’s lock under concurrent HTTP clients, which would show the queue time growing (it was zero in every row above, with one client). It also shows why a lock is not the answer to concurrency: the engine serves one request at a time, and every other request waits for the whole generation. What to do about that is the subject of the next two chapters.

What the numbers do not show

• One model, one worker. GPT-2 small behind one process on a shared RTX 2060. A larger model changes the per-token cost and so the ratio of interface to engine; the interface costs (under 2 ms) are the ones that should transfer.
• Windows 11, loopback, uvicorn, httpx. The localhost delay, the connection reset on a large body and the figures of the transport are properties of this operating system and these libraries, measured on one machine. A Linux host or another client would probably differ on all three.
• A miniature of the schema. The classifier was checked against 19 changes I thought of, on a four-field request and a four-field response. It agreed with behaviour on all 19, which says the rule is right for those; it does not say the rule is complete (it has nothing to say about nested arrays of objects, oneOf, or semantic changes that no schema shows).
• The probe list is mine. Twenty-nine requests are a sample of the ways a client can be wrong. They found four 500s in the first draft; a different list would find different ones.
• Run-to-run variation of 15% in the generation times on this shared GPU. The tables state the run reported; the earlier clean run is in the results file.
• Sequential clients only. One client in the lifecycle table. Queueing, fairness, admission under load and streaming are not measured here.
• Authentication, TLS, a reverse proxy, client disconnects and timeouts were not built or measured.
• The race was measured on the backend in threads, not through HTTP.

Inference Lab increment

The Lab now has a service. build_app takes a backend and Limits and returns the FastAPI application; hardened=False returns the first-draft variant for comparison. Its pieces:

• GenerateRequest, GenerateResponse (version 1) and GenerateResponseV1_1: the contract as pydantic models.
• check_budget and ignored_parameter_warnings: pure functions that return the parameter at fault, or the warning, with the numbers.
• ApiError, envelope and the handlers that put every failure into it, with a request id.
• Stamp (request id, start time, Server-Timing) and BodyLimit (413 by declared or counted size) as ASGI middleware.
• diff_schemas: compatible or breaking, for requests and responses.
• ServerProc, a context manager that runs the service in another process so requests cross a real socket.
• Commands budget, errors, compat, body, fidelity, lifecycle, blocking, race and serve.

To run the service and send it a request (use the address, not localhost):

python examples/ch25/service_lab.py serve --device cuda --port 8000

curl -s http://127.0.0.1:8000/v1/generate -H "content-type: application/json" -d '{"prompt": "The capital of France is", "max_new_tokens": 8}'

The rule this adds: every decision the boundary makes is a named rule that refuses with the parameter and the numbers, the response says what was run and under which identity, and the schema changes only by containment. The original toy service, examples/ch25/api.py, is unchanged and still passes its tests; it is what a first draft looks like before any of the above.

What you should now be able to do

• Break a request’s latency into stages from the response and the headers, and say how much belongs to the interface and how much to the engine.
• Show, by measuring tokens per character on a handful of text kinds, that a character limit cannot stand in for a token budget, and write the check that can.
• Choose a status and a stable error type for each failure a service can produce, and show that the framework’s defaults are or are not the choice.
• Classify a schema change as compatible or breaking for requests and for responses, and say which reader the answer assumes.
• Compare what a service returns with what the engine computes, and say what the wire dropped.

Key terms

• Request lifecycle: the stages from arrival to response; each can fail and each costs time.
• Token budget: the limits on prompt tokens and total positions, enforced after tokenisation and before work.
• Error envelope: one JSON shape for every failure, with a stable type, a message, the parameter at fault and a request id.
• Liveness and readiness: the process answers, against the model is loaded and the service will accept work.
• Tolerant reader: a client that ignores fields it does not recognise.
• Containment (of a schema change): the new schema accepts, or produces, a subset or superset of the old, as the direction requires.
• Request id: an identifier in the response body and header (and the log) that ties the three together.
• Canonical tokenisation: the tokeniser’s own encoding of a text, which a generated sequence need not equal.

Reader exercises

1. Make Limits.context_window come from the Chapter 24 fingerprint of the loaded model, and refuse a request that names a different model.
2. Add Retry-After and a 429 for a full queue, with a bound of your choice, and test it with the lifecycle command at two clients. (Chapter 26 designs the policy.)
3. Add an Idempotency-Key header so a retried request returns the stored response and does not generate again. What must be stored?
4. Write the contract test that fails when a response field’s type changes, using diff_schemas on the generated OpenAPI schema.
5. Return the text as a list of {token_id, text} pieces and decide what the stop-string holdback should do with the pieces.
6. Repeat the probe table against a different server and a different client, and compare the totals.
7. Find out why seven of the 40 sampled responses did not round-trip, and report whether the hypothesis in the chapter holds.
8. Put a reverse proxy in front with a body limit and an Expect: 100-continue handshake, and repeat the large-body measurement. What does the client see now?

References

• RFC 9110: HTTP Semantics — Internet Standard; status codes 400, 413, 415, 422 and 503, and Retry-After; read on 22 October 2026 (summaries of the 413, 415, 422 and 503 sections).
• FastAPI: Handling Errors — framework documentation of RequestValidationError and its replaceable handler; read on 22 October 2026.
• Server Timing — W3C Working Draft, 7 April 2026; the Server-Timing header syntax; read on 22 October 2026.
• Claude API errors and Messages API — provider documentation of an interface (error shape, status codes, request ids), not of the provider’s implementation; read on 22 October 2026. The Messages response fields were not read.
• OpenAPI Specification — industry specification; not re-read for this chapter; the classifier works on pydantic’s JSON Schema, not on OpenAPI.

Evidence in this chapter

Implementation status

• Manuscript: content/books/inference/25-chapter.md
• Runnable examples: examples/ch25/api.py (original toy service, unchanged), examples/ch25/service_lab.py (the service, the schema classifier and the experiments)
• Tests: tests/test_ch25_api.py (toy service), tests/test_ch25_service_lab.py (27 tests: the budget rules; the response shape, request id and Server-Timing; equality with the engine called directly; seeded sampling; every error as one envelope with a request id; the parameter named in a validation error; no echo of the input; the budget checked before generation; the body limit by declared and by counted size; an unhandled failure as a 500 that leaks nothing; liveness separate from readiness; the warning for an ignored parameter; stop strings; the looser behaviour and the echo of the first-draft variant; the classifier’s rules for requests and responses, nested objects, and the real version 1 to 1.1 change; CPU only, with the Chapter 1 table backend)
• Evidence: research/ch25/sources.md, research/ch25/results.json, research/evidence-register.md
• Verification command: python -m unittest discover -s tests -v

The chapter is a draft with draft = false. It makes no claim about any other server, client, model or operating system, about behaviour under concurrent load, or about the safety of the service on a network. Chapter 26 takes the next step: what a service should do when requests arrive faster than the engine can serve them.