The JSON Event Stream

You piped pi --mode json into a log and now you have four thousand lines and a question: which ones are the answer?

The event stream is the canonical reference for everything JSON and RPC mode share, and json.md is unusually honest about one thing — it is a wire format, deliberately reduced from the in-process one. If you build on it without reading that reduction, your reconstruction will drift. This chapter takes the stream apart: the framing, the vocabulary, and the one rule that governs reconstruction.

Framing is not negotiable

json.md states this in the first paragraph of the framing section, and it is the sort of rule that looks pedantic until it corrupts a run:

The stream uses strict JSONL framing. Each record is one JSON object terminated by LF (\n). Split records only on LF and strip an optional preceding carriage return. Unicode line and paragraph separators are valid inside JSON strings and are not record boundaries.

That last clause has a named victim. The documentation says Node.js readline “is not suitable for this stream because it also recognizes those Unicode separators” — U+2028 and U+2029 are legal inside a JSON string, and a line-oriented reader will split a record in half. Use a byte or UTF-8 stream decoder and split on LF yourself.

rpc.md repeats the same rule for the bidirectional case and adds: “Do not use a generic line reader that treats Unicode line or paragraph separators as record boundaries.”

Two operational rules follow. Strip an optional preceding carriage return so CRLF input parses, and read stdout continuously — “a reader that stops consuming records can stall Pi when the pipe buffer fills.” Stdout is reserved for JSONL; diagnostics go to stderr.

The session header comes first

The first record in JSON mode is the session header, and it is shaped like the persisted one:

{"type":"session","version":3,"id":"uuid","timestamp":"2024-12-03T14:00:00.000Z","cwd":"/path"}

RPC mode does not emit it. json.md is explicit: “RPC mode does not emit this record. Use get_state for its current session ID and file.” That is the first structural difference between the two protocols, and it exists because RPC is bidirectional and long-lived — there is no single start moment to attach a header to.

The event vocabulary

json.md groups the events, and the grouping is worth learning as a shape rather than as a list.

Run lifecycle. agent_start (no fields), agent_end (messages, willRetry), agent_settled (no fields), turn_start, turn_end (message, toolResults). A turn is “one assistant response plus any tool calls and tool results produced by that response.”

The distinction between agent_end and agent_settled is the single most important thing to get right, and the documentation repeats it twice. agent_end closes one low-level agent run, and “Automatic retry, overflow recovery, compaction retry, steering, or follow-up work can still continue.” agent_settled means “Pi will not continue automatically through retries, compaction recovery, or queued messages.” If you wait for the wrong one, you will either hang or truncate.

Messages. message_start (message), message_update (usage, assistantMessageEvent), message_end (message). message_end “is the authoritative final message.”

Tool execution. tool_execution_start (toolCallId, toolName, args), tool_execution_update (adds partialResult), tool_execution_end (toolCallId, toolName, result, isError). json.md warns that “Whether it replaces or extends an earlier update depends on that tool’s result contract” — the same caveat that appears in the extensions chapter for partial results.

Queue and state. queue_update (steering, followUp, both complete queues), entry_appended (entry), session_info_changed (name), thinking_level_changed (level).

Compaction. compaction_start with reason of "manual", "threshold", or "overflow", then compaction_end. On success the result carries summary, firstKeptEntryId, tokensBefore, estimatedTokensAfter, usage, and details. On abort, result is absent and aborted is true; on failure, result is absent, aborted is false, and errorMessage describes it. Successful overflow recovery sets willRetry to true.

Retries. auto_retry_start (attempt, maxAttempts, delayMs, errorMessage) and auto_retry_end (success, attempt), plus a separate family for summarisation: summarization_retry_scheduled, summarization_retry_attempt_start (source, reason), summarization_retry_finished.

Here is the sequence for a basic run, straight from json.md:

pi --mode json "List files" 2>/dev/null | jq -c 'select(.type == "message_end")'

That one-liner is the chapter’s practical payload: filter for the authoritative messages and ignore everything else.

Reconstruction: deltas, then authority

This is the part that changes how you write a consumer. json.md states:

Wire message_update records are delta-only. They omit the SDK event’s cumulative message field and every assistantMessageEvent.partial snapshot so stream size remains linear.

In process, the SDK’s AgentSessionEvent carries cumulative snapshots. On the wire, they are gone — “so stream size remains linear.” If you port an in-process subscriber to a JSON consumer and expect partial to be there, it will be undefined.

The nested assistantMessageEvent vocabulary is:

The reconstruction rule, in the documentation’s own terms: use contentIndex to identify the content block; buffer delta fields for a live display; then “replace reconstructed data with the completed content in text_end, thinking_end, or toolcall_end”; and “replace the whole partial message with message_end.message when it arrives.”

The rule is assemble-then-replace, and the levels are ordered.

    flowchart TD
  U["text_delta"] --> D["1 buffered deltas"]
  X["..._end"] --> C["2 authoritative content"]
  M["message_end"] --> A["3 authoritative message"]
  D -->|"replaced by"| C
  C -->|"replaced by"| A
  

Three record kinds, three levels of authority, increasing. The ..._end row is text_end, thinking_end, or toolcall_end, and level 1 is a display convenience, not a source of truth.

One subtlety worth knowing: start, done, and error remain in the type union, but “the normal agent loop translates provider-level start, done, and error into message_start and message_end session events rather than emitting them as message_update.” A consumer that handles them is correct; a consumer that expects them is wrong.

And usage is cumulative but lazy: the top-level usage is “the latest cumulative provider-reported usage for the assistant response,” which “can remain zero until completion when a provider does not report usage while streaming.” Do not bill from it mid-stream.

The TypeScript mirror

json.md publishes the exact transformation, which is worth reading because it names precisely what was removed:

type JsonAgentSessionEvent =
  | Exclude<AgentSessionEvent, { type: "message_update" }>
  | {
      type: "message_update";
      usage: Usage;
      assistantMessageEvent: JsonAssistantMessageEvent<AssistantMessageEvent>;
    };

Everything except message_update passes through unchanged. Only message_update is transformed, and only by stripping partial. The exported JsonAgentSessionEvent type is available from @earendil-works/pi-coding-agent for consumers that want the contract in their own types.

What is not in the JSON stream

Two absences to plan around.

Extension UI records are not session events. json.md states they “are a separate RPC subprotocol, not AgentSessionEvent values.” An extension calling ctx.ui.confirm() in JSON mode does not produce a record you can answer — there is nothing to answer it with.

Two RPC-only events exist. bash_execution_update streams one record per output chunk for a direct RPC bash command, with an optional id matching the command. And extension_error appears when an extension handler throws:

{"type":"extension_error","extensionPath":"/path/to/extension.ts","event":"tool_call","error":"Error message"}

Neither appears in JSON mode. If your automation depends on knowing that an extension failed, you need RPC or the in-process API — a one-shot JSON run will not tell you.

Contrast with the chapter before

Print mode gave you a string and an exit code, and told you nothing about how the answer was produced. The JSON stream gives you everything: every turn, every tool call, every token delta, every retry. What it costs is that you now own correctness. Print mode’s contract was “read to end of stream and trust the exit code”; this one is “split on LF, buffer deltas, replace with authoritative content, replace again with message_end.message, and wait for agent_settled rather than agent_end.”

The next chapter keeps the same event vocabulary and removes the process boundary entirely.

Next: SDK Sessions — the same agent, in your process, with typed methods instead of parsed strings.