Cold Starts, Warm Runs and Real Latency

“Local AI is faster” is not a measurement.

It compresses several different waiting periods into one adjective. A browser-native feature can avoid network round trips and still make a user wait for model acquisition, process startup, session creation, context ingestion or slow generation.

To understand latency, we have to take the lifecycle apart.

1. One duration hides several clocks

For one interaction, useful timestamps include:

The perceived wait for a streaming feature is often dominated by time to first output:

2. The first real observation measured only acquisition

Our first exported Chrome trace contained ten events. It established:

• LanguageModel was exposed;
• availability returned available;
• session creation completed in approximately 8.2 milliseconds;
• the session reported quota and lifecycle capabilities;
• no prompt was executed.

Therefore the trace says nothing about prompt latency.

This sounds obvious, but it is a common measurement error. A fast session constructor is not a fast answer. A fast first chunk is not a fast completion. A completed generation is not a successful feature.

The missing values in the report are correct:

Absence is better than a number inferred from the wrong event.

3. A progress callback is not proof of a download

The same trace contained two downloadprogress callbacks:

loaded = 0
loaded = 1

They arrived roughly 0.7 milliseconds apart while capability state was already available.

The callback name comes from the API. Interpreting it requires the surrounding state. This event sequence cannot reasonably establish that a complete model asset was transferred during those 0.7 milliseconds.

The supported conclusion is narrower:

The session monitor emitted acquisition-progress callbacks during creation.

It may be notifying listeners of an already-complete resource state. It may represent another browser-managed step. Without network or browser-internal evidence, the Observatory should not relabel it as bytes downloaded.

This is why traces need events rather than prose such as “model downloaded successfully.”

4. Cold and warm are operational definitions

“Cold” can mean several things:

• model absent from disk;
• model installed but process not started;
• browser restarted;
• session absent;
• OS file cache empty;
• accelerator context absent.

An extension cannot reliably force or verify all of them.

Our run protocol therefore uses explicit operational labels:

• Download path: availability indicates acquisition and progress is observed.
• Cold start: first session and prompt after browser restart; the model may already be installed.
• Warm session: the identical prompt runs again in the same live session after only the trace is cleared.

These labels say what we did. They do not claim complete control of hidden caches.

5. Compare paths, not isolated numbers

For matched runs, useful deltas include:

• browser build and flags;
• hardware and power state;
• fixture input and creation options;
• capture mode;
• competing workload;
• repetition count;
• whether the session is reused or recreated.

Even then, three repetitions produce an initial estimate, not a performance law.

The report uses medians because one startup interruption can dominate a mean in a tiny sample. We should still retain every raw observation, including failures and slow outliers.

6. Throughput needs a denominator we can defend

Streaming produces chunks, but chunks are not tokens. One implementation may emit one token per chunk; another may batch several; another may revise or replace accumulated text.

The Observatory can safely report:

• chunk count;
• output characters and bytes;
• time to first chunk;
• total duration;
• character throughput as an interface-level approximation.

It should not report tokens per second unless the runtime supplies a reliable token count or we explicitly label a tokenizer-based estimate.

For output characters $C$ after the first chunk and generation interval $G$:

7. Streaming changes cancellation

A long generation creates an opportunity to cancel:

const controller = new AbortController();

const result = session.promptStreaming(input, {
  signal: controller.signal
});

stopButton.addEventListener("click", () => controller.abort());

Cancellation latency matters too:

user clicks Stop
      ↓
abort requested
      ↓
stream ends
      ↓
session state observed

We need to measure whether partial output remains visible, whether the session is reusable, and whether usage increases after an aborted operation.

An AbortError is an expected outcome when the user stops work. It should not be counted as a model failure.

8. Benchmark the feature boundary

A local generation can be fast while the user-facing feature remains slow because of:

• DOM extraction;
• document segmentation;
• prompt construction;
• model acquisition;
• output parsing;
• validation;
• rendering;
• application-side persistence.

The Observatory’s direct prompt trace measures the runtime boundary. Its application SDK can add the surrounding feature events.

selection captured
    ↓
context prepared
    ↓
prompt started
    ↓
first chunk
    ↓
prompt finished
    ↓
validation finished
    ↓
UI committed

Only then can we explain what the user waited for.

9. Model changes require paired experiments

The motivating experiment compares the same application boundary under different browser-managed configurations.

The Observatory records the model/configuration label as operator-supplied because the API does not attest a model identity to the extension. A report can therefore say:

Under the operator-recorded “Gemma 4 flag enabled” configuration on Chrome 152.0.7977.65, this run produced the following measurements.

It cannot say:

Chrome proved that Gemma 4 generated this output.

This distinction may feel cautious, but it is what makes later comparisons credible.

10. The next corpus

The minimum useful performance corpus is now defined:

1. one cold Prompt fixture run;
2. one identical warm-session run;
3. one cloned-session continuation;
4. one explicit cancellation;
5. three repetitions for any latency claim we publish;
6. the same matrix under the comparison configuration;
7. raw failures retained beside successful runs.

The chapter can explain the measurement system now. Its numerical comparison remains open until those traces exist.

That is not a weakness. It is a visible experimental boundary.

Conclusion

Browser AI latency is a lifecycle, not a stopwatch.

Capability inspection, acquisition, session creation, first output, completion and validation answer different questions. Progress callbacks require context. Cold and warm require operational definitions. Chunk count must not masquerade as token count.

Our first real trace measured a fast session creation and nothing about inference. The instrument correctly left the missing fields empty.

The next chapter asks why the same session configuration could work and then fail. Once the browser manages the model, model lifecycle and compatibility become part of application engineering even when the application never downloads a model itself.

Sources and further reading

1. Chrome for Developers, The Prompt API.
2. Chrome for Developers, Get started with built-in AI.
3. Chrome for Developers, Understand built-in model management in Chrome.