Part IX — Production Debugging and Prevention

The refund that nobody can replay

Part VIII built machine-assisted debugging machinery: contracted traces, hypothesis generation under evidence rules, discriminating experiments checked by intervention outcomes. That machinery means nothing without production observability. A practitioner inherits exactly this: a support agent double-refunds one customer on Tuesday, the on-call engineer asks what the model saw, and the answer is a dashboard screenshot of the output text plus a log line reading POST /chat 200 812ms. No prompt hash, no retrieved documents, no parameters, no seed. The incident is real, the evidence is gone, and every diagnosis from here on is storytelling.

OBSERVATION: the production log for the incident window holds output text and latency only; the assembled model input, retrieval snapshot, model identifier, and sampling parameters are absent for the failing request. HYPOTHESIS H1 (emittable but unemitted): the evidence needed for replay existed at request time and was discarded by the logging design. H2 (never present): the serving path never had the evidence available to emit. H3 (emitted but unfindable): the evidence was logged somewhere but cannot be joined to the failing request. INFERENCE: none yet — H1/H2/H3 predict different emission-point signatures and separate only once the emission contract states what every request must carry.

This chapter’s question: what must a production AI system emit so that any failure can be detected, localized, and replayed — and what sampling and retention policy keeps that affordable?

Why “log the prompt and the answer” fails first

The obvious move — logging prompt text and completion text — fails because an AI request is not two strings. It is an assembled build with an environment, and text-only logs discard the build. Six defects hide behind text logging:

1. No identity. Two “same prompt” requests differ in retrieved chunks, history truncation, or a silent model-version roll. Without hashes and pins, sameness is an OPINION, not a MEASUREMENT.
2. No join key. Prompt logs live in the app, token counts in the gateway, retrieval logs in the search service. Without one request ID spanning all three, the failing request cannot be reassembled.
3. No parameters. Temperature, top-p, max-tokens, seed, and stop sequences change the output distribution under a fixed prompt. A log without them cannot replay.
4. No retrieval snapshot. The index drifts daily; Tuesday’s chunks differ from Monday’s. Output logged without the retrieved document hashes is evidence against a context that no longer exists.
5. No cost/latency segments. One 812ms number never says whether retrieval, guardrail, model, or tool calls spent it. Unsegmented timing cannot attribute delay or spend — Chapter 55 owns that triage, but this chapter must emit its inputs.
6. Full-capture bankruptcy. Logging every full context forever at high traffic costs more than the incident it would solve. No sampling or retention policy means observability gets deleted by finance instead of designed by engineering.

OPINION: a production AI system that logs only text is a crime scene cleaned before the detective arrives. Emit the build, not the souvenir.

The mental model: the per-request record — one joinable bundle per inference carrying identity (request ID, hashes), environment pins, parameters, retrieval snapshot, segmented timing and spend, outputs, and verdicts. Sampling decides how many bundles are kept whole; retention decides how long each field class survives. Everything downstream — replay (Ch53), guardrails (Ch54), cost triage (Ch55), live response (Ch56) — consumes this bundle or declares UNKNOWN.

This record is a wide event in the observability-engineering sense: an arbitrarily wide, high-cardinality, structured row per unit of work. Majors, Fong-Jones, and Miranda draw the line as monitoring answers known-unknowns with pre-decided metrics, while observability answers unknown-unknowns — and the only way to answer a question you have not thought of yet is to have captured the context before you needed it, without shipping new code (Majors, Fong-Jones & Miranda, 2022). Text-only logs are the narrow-event failure: they answer “what did it say” and nothing else. The request ID that joins app, gateway, and retrieval is a trace ID in the Dapper sense (Chapter 37) — and the per-request record itself is Chapter 37’s trace object emitted in production: the same span identity, annotations, and causal edges, differing only in emission point (the serving path, on every request) and retention (tiered, below).

The method: emission contract, sampling policy, retention schedule

Every production AI request emits a per-request record with six field classes, joined by one request ID:

1. Identity. Request ID, timestamp, caller/service, prompt hashes per role (system/task/guardrail), assembled-input hash, output hash. The hash decides what ran; prose resemblance decides nothing.
2. Environment pins. Model identifier as logged by the serving path, retrieval index snapshot, code/config revision, feature flags. A prompt hash plus an unpinned environment is half an identity (Ch30’s rule, enforced here at runtime).
3. Parameters. Temperature, top-p/top-k, max tokens, seed where supported, stop sequences. Logged verbatim — never “defaults,” because defaults drift silently.
4. Retrieval and tool snapshot. Retrieved document IDs plus content hashes, tool calls with arguments and results hashes, truncation record (what was cut, where). Enough to re-assemble the exact model input or to mark re-assembly UNKNOWN.
5. Segmented telemetry. Per-stage latency (retrieval, assembly, model, guardrail, tools) and per-stage token/cost counts. Totals alone are explicitly insufficient.
6. Verdicts. Guardrail decisions, refusal flags, user feedback events when present — recorded as events, never as diagnoses.

    flowchart TD
    RQ["every production AI request"] --> E["emit a per-request record: identity + env pins + parameters + retrieval / tool snapshot + segmented telemetry + verdicts"]
    E --> J["join across app / gateway / retrieval / tools by ONE request ID (W3C trace context)"]
    J --> T{"retention tier"}
    T -->|"Tier A"| TA["headers always: hashes, pins, params, segments, verdicts"]
    T -->|"Tier B"| TB["full inputs / outputs for a declared fraction + 100% of failures, guardrail trips, flagged turns (deterministic sample)"]
    T -->|"Tier C"| TC["payloads age into counts + distributions; hashes + segments survive"]
    TA --> I["on incident: load the record, recompute the assembled-input hash"]
    TB --> I
    I --> M{"re-assembly hash matches?"}
    M -->|yes| REP["REPLAYABLE — route to the Ch53 conveyor, >=3 replay trials"]
    M -->|"no / field absent"| UNK["DEGRADED or UNKNOWN — name the missing field, never backfill from memory"]
  

PER-REQUEST RECORD (frozen schema v1):
request_id: r-7f31 | ts: ___ | prompt hashes: sys@___ task@___ guard@___
assembled hash: b3:___ | model: <as logged> | index: idx-___ | params: <verbatim>
retrieved: [doc-__ hash ___, ...] | truncated: ___ bytes at ___ | tools: ___ (hash ___)
segments: retr ___ms / model ___ms / guard ___ms / tools ___ms | tokens: in ___ out ___
output hash: ___ | verdicts: ___ | UNKNOWN fields: ___ (named, not silent)
RULE: any field absent at read time is UNKNOWN for that request — never backfilled from memory.

Sampling and retention make the contract affordable. The policy has three tiers, declared in advance and marked as setup choices, not universal optima:

• Tier A (always kept): identity, pins, parameters, hashes, segments, verdicts — small, joinable, retained longest (e.g., a duration the team declares; the number is a setup choice, recorded in the config, changeable as traffic changes).
• Tier B (sampled whole): full assembled inputs and outputs kept for a declared fraction of traffic plus 100% of failures, guardrail trips, and user-flagged turns. Sampling is deterministic (hash of request ID), never “keep the interesting ones” by gut feel. This is the standard two-part pattern: head-based deterministic sampling for the baseline fraction, plus tail-based retention that inspects the completed request and keeps every error, timeout, and flagged turn — a combination that cuts trace volume by most of its bulk while losing no failures (Kaldor et al., 2017).
• Tier C (aggregates only): full-text payloads age into counts and distributions after a declared window; hashes and segments survive. Re-assembly after Tier C expiry is UNKNOWN by policy, not by accident.

OBSERVATION (constructed illustration, not a measured run): the double-refund request carries assembled hash b3:81aa with index idx-2026-08-14 and Tier B full capture because the guardrail tripped; the three requests before it carry Tier A headers only, joinable by request ID but not re-assemblable. UPDATED BELIEF: H1 supported for this instance (the evidence was emittable; the old design discarded it); H2 exonerated here (serving path had all fields); H3 retired going forward (one join key ends the scatter). No claim about optimal sampling fractions — those are local cost decisions.

No model-generated summary of the incident (“the model was confused by the policy wording”), no confidence value, no agreement across retries, and no downstream symptom (“the customer stopped complaining”) substitutes for the per-request record. Records replay; narration decorates.

Example: the double-refund, now replayable

The practitioner replays Tuesday’s double authorization from the record instead of from memory:

# observability-first replay: records before theories (no repair yet)
rec = load_request("r-7f31")  # OBSERVATION: full Tier B bundle, schema v1
assert rec.assembled_hash == recompute(rec.prompt, rec.retrieved, rec.history)
# MEASUREMENT: hash matches -> input re-assembled bit-exact; else UNKNOWN, stop here
env = pin_environment(rec.model, rec.index_snapshot, rec.params)  # fixed pins
for trial in range(3):  # nondeterminism gets trials, not single samples
    out = rerun(rec.assembled_input, env)  # predictions pre-written per H1/H2/H3
    log(trial, hash_output(out), segments(out))
# Route: same failure under identical bundle -> downstream chapters own cause;
# divergent failure -> environment or sampling suspect; missing fields -> UNKNOWN.

In the constructed case the re-assembly hash matches, the three replay trials reproduce the double authorization twice and refuse once — a distribution, honestly reported — and the record shows the guardrail verdict field empty: no guardrail evaluated this path at all. The finding is not a cause; it is an instrumentation verdict that directs Chapters 53–54: the failure is replayable, and the guardrail gap is now MEASUREMENT rather than suspicion. The licensed claim covers this request under this schema — not production AI systems in general.

Research lineage: wide events, trace context, and a GenAI schema

The schema is being standardized — up to a point. OpenTelemetry’s GenAI semantic conventions (Chapter 37) define span and event attributes for LLM and agent calls — model identity, parameters, token counts, tool calls, and a privacy-gated content-capture toggle. By 2026 they are CNCF-backed, emitted natively by agent frameworks (LangChain, CrewAI, AutoGen), and ingested by the major tracing vendors, so adopting them rather than inventing a local schema means the per-request record joins the same backend as the rest of the stack. They cover field classes 1–5. They do not yet standardize class 6: guardrail decisions, refusal flags, and evaluation verdicts sit outside the conventions’ current scope — the GenAI working group lists output-quality and safety scoring as in progress — so the Verdicts field stays a local extension layered on top of the standard spans, and it is exactly what Chapters 54 and 49 populate.

Trace context propagation is a solved problem. The reason one request ID can span the app, the gateway, the retrieval service, and the tool calls is W3C Trace Context: a traceparent header carried through every hop. The “join-key scatter” failure mode is a system that logs but does not propagate — a fixed problem with a standard fix.

Retention is a cost-engineering decision, and it belongs to engineering. Observability-engineering practice is explicit that the answer to “we cannot afford to keep everything” is a designed sampling and retention policy — deterministic, tail-aware, tiered — not an undesigned cap imposed later by a storage bill (Majors, Fong-Jones & Miranda, 2022). The three tiers in this chapter are that policy with the numbers left to the team. Tier B’s tail — “keep every failure” — also has a research literature that makes it smarter than a flat error filter: retroactive sampling keeps trace generation always-on and cheap and hydrates the full bundle only when a trigger fires (Zhang et al., 2023), while anomaly-biased and dissimilarity-preserving selection (Sifter, SoCC 2019; STEAM, ESEC/FSE 2023) beat a flat fraction for the healthy-baseline half. None of them changes the head-based baseline or the rule that failures are kept whole.

Lab 52: emission-gap audit with pre-written replay predictions (proposed)

PROPOSED, not executed: no author-measured results are reported. The evidence this chapter requires is the reader’s own production log.

Setup. Pick one production AI endpoint you own. Freeze one week of traffic metadata. The record completeness (Tier A headers vs. full Tier B bundle) is the independent variable; endpoint, task, and traffic mix are controlled.

Task.

1. Before auditing, write H1/H2/H3 with distinct predicted signatures: H1: “fields existed at request time but were discarded (present in code path, absent in logs)”; H2: “fields never available in the serving path (no code emits them)”; H3: “fields logged but unjoinable (no shared request ID).”
2. Sample 30 failing or flagged requests and 30 clean ones; for each, attempt re-assembly of the exact model input from logs alone, ≥3 replay trials where re-assembly succeeds.
3. Record per-request verdict: REPLAYABLE / DEGRADED (Tier A only) / UNKNOWN, with the missing field named.

Success criterion. A per-request table with join verdicts, named UNKNOWN fields, and replay-trial counts. A new logging library installed but unmeasured is explicitly not completion.

Companion tool: Production Observability Inspector

What it accepts: the declared emission schema, a sample of per-request records, the serving-path code that emits them, and the sampling/retention config. What it performs: it checks every sampled record against the schema field-by-field, verifies joinability across services by request ID, attempts input re-assembly and reports hash match/mismatch, audits that sampling is deterministic and failure-inclusive, and names every UNKNOWN field instead of silently passing. What it can establish: whether a given request is replayable, degraded, or UNKNOWN — and which field or join is missing — for the examined sample only. What it cannot establish: causes of the underlying failure, optimal sampling fractions, or future log sufficiency. It never treats model self-explanations, confidence values, agreement across retries, single-run outcomes, or downstream symptoms as observability evidence. How its output changes your next action: REPLAYABLE routes to Chapter 53’s conveyor; missing-field verdicts route to emission repair (add the emitter, add the join key, widen Tier B); policy verdicts route to sampling/retention revision — each as one intervention with pre-written predictions.

Paper form, sufficient for this chapter:

Endpoint: ___  Schema: v___  Sample: ___ failed + ___ clean
REPLAYABLE ___ / DEGRADED ___ / UNKNOWN ___ (missing: ___)
Join: ok / broken at ___  Sampling: deterministic? ___ failure-inclusive? ___
NEXT REPAIR: emitter / join key / Tier B width / retention

Where a software implementation does not yet exist in the reader’s stack, this record is the tool. Emit before explaining.

Reusable procedure: instrument every AI endpoint for replay

1. Declare the schema — six field classes, one request ID, versioned.
2. Emit at the serving path — headers always, full bundles per sampling policy.
3. Join across services — one ID spanning app, gateway, retrieval, tools.
4. Audit by re-assembly — hash-match decides replayability, nothing else.
5. Publish sampling/retention — tiers declared, deterministic, failure-inclusive.

Failure modes

• Text-only logging. Prompt and answer saved, build discarded. Souvenirs, not evidence.
• Join-key scatter. Three services, three IDs. The request exists everywhere and nowhere.
• “Defaults” parameters. Unlogged sampling settings that drift. Replay runs a different program.
• Snapshot amnesia. Output kept, retrieved context forgotten. Evidence against a vanished context.
• Total-only timing. One latency number for five stages. Attribution impossible — starves Chapter 55.
• Curated sampling. Keeping “interesting” requests by feel. Bias laundered as policy; failures missed.
• Silent UNKNOWN. Absent fields treated as “probably fine.” Name the gap or inherit it mid-incident.
• Retrospective backfill. Reconstructing inputs from memory after the incident. Memory is not a log.

Limits, per contract: one inspection covers the examined endpoint, schema version, and sample window; it does not certify causes, does not transfer across endpoints, and does not bless any sampling fraction as sufficient. UNKNOWN wherever fields are absent, IDs unjoined, or payloads aged out by policy.

References

• Charity Majors, Liz Fong-Jones, and George Miranda. Observability Engineering: Achieving Production Excellence. O’Reilly Media, 2022. https://www.oreilly.com/library/view/observability-engineering/9781492076438/
• Jonathan Kaldor, Jonathan Mace, Michał Bejda, Edison Gao, Wiktor Kuropatwa, et al. Canopy: An End-to-End Performance Tracing and Analysis System. Proceedings of the 26th ACM Symposium on Operating Systems Principles (SOSP), 2017, pp. 34–50. https://doi.org/10.1145/3132747.3132749
• Benjamin H. Sigelman et al. Dapper, a Large-Scale Distributed Systems Tracing Infrastructure. Google Technical Report, 2010. https://research.google/pubs/dapper-a-large-scale-distributed-systems-tracing-infrastructure/
• OpenTelemetry Authors. Semantic Conventions for Generative AI Systems and W3C Trace Context. OpenTelemetry / W3C, 2024– (CNCF-backed, evolving; output-quality and safety scoring in progress as of 2026). https://opentelemetry.io/docs/specs/semconv/gen-ai/
• Lei Zhang, Zhiqiang Xie, Vaastav Anand, Ymir Vigfusson, and Jonathan Mace. The Benefit of Hindsight: Tracing Edge-Cases in Distributed Systems. 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI ‘23), 2023. https://www.usenix.org/conference/nsdi23/presentation/zhang-lei

Debugging Checklist

• Per-request schema declared and versioned (six field classes)?
• One request ID joins app, gateway, retrieval, and tools?
• Prompt hashes, assembled-input hash, and output hash logged per request?
• Model, index snapshot, revision, and verbatim parameters pinned?
• Retrieval hashes, tool hashes, and truncation record present or named UNKNOWN?
• Per-stage latency and token/cost segments (not totals only)?
• Sampling deterministic, declared, and failure-inclusive?
• Retention tiers declared with changeable durations recorded as setup choices?
• Re-assembly hash-checked on samples with ≥3 replay trials?
• No self-explanation, confidence, agreement, single runs, or symptoms cited as evidence?

What This Chapter Established

• The per-request record as the emission contract: identity, pins, parameters, retrieval/tool snapshots, segmented telemetry, verdicts — demonstrated on the constructed double-refund replay, no measured runs claimed.
• The three-tier sampling/retention policy as the affordability mechanism, with durations and fractions marked as local setup choices.
• Lab 52 as a proposed emission-gap audit the reader executes; the Production Observability Inspector contract (accepts/performs/can-establish/cannot-establish/next-action).
• What was NOT proved: any causal claim about the refund failure, any optimal sampling fraction, or any cross-endpoint generality. One contract declared; nothing universal.
• Research grounding: the per-request record is a wide event — observability answers unknown-unknowns, which requires capturing context before you need it (Majors et al.) — and it is Chapter 37’s trace object emitted in production, differing only in emission point and retention; the request ID is a Dapper-style trace ID propagated via W3C Trace Context, and field classes 1–5 map to OpenTelemetry’s GenAI conventions while class 6 (verdicts) stays a local extension the conventions do not yet cover; Tier B is head-based deterministic sampling plus tail-based error retention (Canopy / Kaldor et al.), with retroactive and anomaly-biased sampling (Hindsight; Sifter; STEAM) as the research frontier for the tail, and retention is a designed engineering policy, not an undesigned storage cap.
• Position in the arc: Part VII routed multi-agent blame across boundaries; Part VIII (Ch44–51) constrains machine debuggers to evidence contracts. This chapter gives both something to consume in production: replayable records or named UNKNOWNs.

Next

Records exist — or their absence is now named. But a replayable record sitting in a log is not prevention: the next incident will still start from a page, a frozen bundle assembled under pressure, a repro built by hand, and a postmortem written from memory unless the path from failure to regression is itself a procedure. Chapter 53, “From Production Failure to Regression,” builds that conveyor; what survives the trip from page to pinned test is its chapter’s to establish, not this one’s.