A Resolved Promise Is Not a Correct Answer

The following code has only one definition of success:

try {
  const output = await session.prompt(input);
  show(output);
} catch (error) {
  showFailure(error);
}

If the promise resolves, the interface displays the result.

That is sufficient for a deterministic lookup whose return value is authoritative. It is not sufficient for a probabilistic system that can return fluent, well-formed and incorrect text.

A resolved promise proves that an operation completed according to the runtime contract. It does not prove that the feature did its job.

1. Success has layers

One browser-AI operation can pass one layer and fail the next:

These are not competing definitions. They form a validation pipeline.

runtime completion
      ↓
structural validation
      ↓
semantic validation
      ↓
feature policy
      ↓
authorized effect

The system should retain the outcome at every boundary.

2. Fluent output is not evidence of correctness

Consider the summarization fixture introduced earlier. It protects a distinction:

browser owns model lifecycle
application owns feature behavior

A summary can be grammatical, brief and convincing while collapsing those responsibilities into one claim. Nothing in promise resolution detects that change.

Likewise, a rewrite can become clearer while weakening “must” into “may.” A translation can sound natural while reversing the actor. A language detector can return a perfectly valid ranked array whose top candidate is wrong.

The failure belongs to the feature contract, not the transport.

3. Assertions make the contract executable

Browser AI Observatory fixtures combine deterministic checks with human rubrics:

assertions: [
  { kind: "contains", value: "browser" },
  { kind: "contains", value: "application" },
  {
    kind: "custom",
    rubric: "Do not claim local execution is automatically private or faster."
  }
]

The automatic checks are cheap and reproducible. The rubric protects a semantic claim that substring matching cannot evaluate honestly.

The evaluator returns three states for individual checks:

• passed: true;
• passed: false;
• passed: null, requiring human review.

It does not convert “not automatically checkable” into a pass.

4. A check can pass for the wrong reason

Suppose an assertion searches for browser. This output passes:

The browser has nothing to do with model management.

The required word appears while the claim is reversed.

Lexical assertions are useful for narrow invariants such as required keys, product names and forbidden phrases. They should not be presented as semantic judges.

Every assertion has a scope:

{
  "kind": "contains",
  "value": "browser",
  "claim": "required term appears",
  "doesNotClaim": "the surrounding statement is correct"
}

Evaluation becomes more trustworthy when each metric states what it does not establish.

5. Preserve the output and the judgment separately

An evaluation event should refer to the prompt trace without rewriting it:

{
  "type": "fixture.evaluated",
  "correlation": {
    "sessionId": "session-7",
    "traceId": "prompt-12"
  },
  "data": {
    "fixtureId": "prompt-session-accounting",
    "passed": true,
    "automaticChecks": 4,
    "humanChecks": 1
  }
}

Later human review can append another judgment with its reviewer, rubric version and decision. It should not mutate the original generated output or pretend the automatic evaluator made the later decision.

This separation enables disagreement and re-evaluation.

6. Failure, rejection and abstention differ

The runtime can fail before producing output. A validator can reject completed output. The feature can also abstain because the evidence is insufficient.

switch (result.outcome) {
  case "runtime-failed":
    return showRetryOrFallback(result.error);
  case "invalid":
    return showRejectedOutput(result.reasons);
  case "abstained":
    return askForMoreInformation(result.reason);
  case "accepted":
    return commitFeatureResult(result.value);
}

Collapsing all four into “AI failed” prevents diagnosis. Collapsing all completed calls into “AI succeeded” is worse.

Abstention can be the correct feature result. A language detector that refuses to guess from two ambiguous characters may be behaving better than one that always chooses a language.

7. Validation has its own latency and failure modes

If a feature performs model generation and then validation, user-visible latency is:

The trace should therefore record prompt.finished before feature.validation.finished. Otherwise a slow validator can be mistaken for slow inference.

The validator must also be constrained. Sending private local output to a remote judge silently would undo the data-route properties that motivated local inference.

8. Use domain invariants before general quality scores

A general score such as “helpfulness: 0.86” is difficult to interpret. Feature-specific invariants are more actionable:

• every cited source exists;
• dates match the input;
• negation is preserved;
• no new permissions are claimed;
• required fields parse;
• an uncertain language result abstains;
• a rewrite changes no protected term.

These checks are incomplete, but failure explains what broke.

The strongest evaluator is often the downstream deterministic system. If generated code must compile, compile it. If a tool argument names a file, verify that file is within the allowed scope. If a summary feeds a decision, test the protected facts the decision requires.

9. Human review needs a visible question

“Does this look good?” is not a rubric.

A useful human check names the decision:

Did the rewrite preserve that the user must grant permission before instrumentation begins?

[Yes] [No] [Unclear]

The reviewer should see the input, output, protected claim and relevant context. They should not need to infer what the fixture author intended.

Human judgment remains fallible. Versioning the rubric and retaining Unclear make that uncertainty visible.

10. The debugger must show two timelines

Operational events answer what ran:

inspect → create → prompt → chunks → completion

Evaluation events answer what passed:

parse → assertions → human rubric → feature decision

The completed Agent Inspector will align them by trace ID. A developer should be able to see that a request completed in 420 milliseconds and was rejected because it changed a permission claim.

That is more useful than a green “200 OK” beside bad output.

Conclusion

A resolved promise is an operational fact.

Correctness requires additional evidence: structural validity, domain invariants, semantic review, feature policy and authority checks. Each layer can accept, reject or abstain for a different reason.

Browser AI Observatory already places fixture evaluation beside runtime traces. The next step is to turn individual fixtures into a repeatable feature evaluation: representative cases, meaningful slices, baselines and release gates.

Sources and further reading

1. Chrome for Developers, The Prompt API.
2. Chrome for Developers, Built-in AI APIs.
3. NIST, Artificial Intelligence Risk Management Framework.