Knowing When Not to Answer

Chapter 10 ended with an ambiguity.

A model gave weak, generic advice.

Perhaps it ignored decisive evidence.

But perhaps the evidence never justified a decisive answer in the first place.

Those are not the same failure.

Consider this question:

What was Company A's Q3 revenue?

and this evidence:

Q1 revenue was $41.2 million.
Q2 revenue was $43.8 million.
The company expects second-half demand to improve.

A language model can easily produce something that sounds reasonable:

Q3 revenue was approximately $46 million.

The estimate may even be numerically plausible.

It is still unsupported.

The correct system-level response is not necessarily another attempt at better generation.

It may be:

The supplied evidence does not contain Q3 revenue.
I can answer Q1 and Q2, but I would need the Q3 filing
or another authoritative source to answer the question.

That response is less impressive as prose.

It is more reliable as a system action.

This chapter develops the missing capability:

A reliable system must know not only how to answer, but whether the available evidence is adequate to justify answering at all.

Call this property epistemic adequacy.

It is not the same thing as confidence.

It is not the same thing as refusal.

It is not the same thing as containment.

It asks whether the information available to the system is sufficient for the answer the system is about to commit to.

Where we are

The book now has several distinct measurements:

CONTAINMENT
Does the claim extend beyond the evidence representation?

STRUCTURAL FIDELITY
Are roles, polarity, quantity, time, and relations preserved?

CONSISTENCY
Does the system preserve what should remain stable?

SENSITIVITY
Does the system respond when decisive context changes?

None answers this question:

Do we have enough evidence to answer?

That question belongs to a different layer.

A claim can be:

well contained
structurally coherent
sensitive to context

and still be based on an incomplete evidence chain.

Conversely, a model can be uncertain even when an authoritative database contains the exact answer.

The system therefore needs to distinguish:

model uncertainty

from:

evidence insufficiency.

And once insufficiency is detected, it must decide whether to:

retrieve
ask
verify
answer partially
abstain
or refuse for policy reasons.

That is the architecture of this chapter.

1. Abstention is not failure

Ordinary benchmark culture rewards answering.

Each question arrives with the implicit instruction:

produce an answer

so an empty answer looks like failure.

Real systems operate under a different objective.

If the evidence is inadequate, answering can be the failure.

This has been recognized in machine learning for decades through classification with a reject option and selective prediction: a system may choose not to predict on examples for which it cannot maintain acceptable risk.

The same idea becomes more important for language models because the model can always generate something.

That is precisely the problem.

The language model does not naturally encounter a hard boundary between:

known
unknown
underspecified
contradictory
stale
unavailable

It encounters another next-token distribution.

So the surrounding system needs an explicit answerability decision.

The ability to continue generating is not evidence that the system should continue asserting.

2. Abstention, refusal, clarification, and deferral are different actions

The phrase “the model refused” collapses several different situations.

We should separate them.

These actions may produce superficially similar language:

I cannot answer that.

But their meanings are completely different.

A system that knows the answer but is prohibited from supplying it is not epistemically uncertain.

A system that lacks a required date should not pretend it encountered a safety restriction.

A user who omitted a critical parameter should usually be asked for that parameter rather than receiving a permanent abstention.

The reason matters because the next action depends on it.

3. Confidence is not epistemic adequacy

A common implementation is:

if model_confidence < threshold:
    abstain
else:
    answer

Sometimes that is useful.

It is not enough.

Model confidence and evidence adequacy answer different questions.

High confidence, inadequate evidence

The model may strongly “know” from pretraining that a company usually reports on a particular date.

But if the task contract says:

answer only from the supplied filings

then parametric confidence is irrelevant.

Low confidence, adequate evidence

An authoritative database may explicitly return:

Q3 revenue = $47.3 million

The model may still be uncertain about how to phrase the answer.

The evidence is adequate even if the generator’s internal confidence is not.

Wrong uncertainty attribution

A query may be ambiguous because the input is incomplete, or difficult because the model lacks capability.

Those cases require different responses.

Recent work on uncertainty attribution makes this distinction explicit: data uncertainty should often trigger clarification, while model uncertainty may call for tools, retrieval, or human help rather than another guess.[1]

So the central design principle is:

Do not ask only whether the model feels uncertain. Ask whether the task’s required evidence and distinctions are actually available.

4. Define an answerability contract

We have used measurement contracts, evaluation contracts, attack contracts, and perturbation contracts throughout the book.

Abstention needs its own contract too.

answerability_contract = {
    "question_type": "reported_financial_value",
    "target": "company_q3_revenue",
    "required_fields": [
        "company_identity",
        "reporting_period",
        "revenue_value",
        "currency",
    ],
    "critical_fields": [
        "reporting_period",
        "revenue_value",
    ],
    "acceptable_sources": [
        "company_filing",
        "audited_financial_statement",
        "authoritative_database",
    ],
    "freshness_requirement": "covers_requested_period",
    "conflict_policy": "must_resolve_before_definitive_answer",
    "partial_answer_allowed": True,
    "recovery_actions": [
        "retrieve_filing",
        "query_financial_database",
        "ask_user_for_source",
    ],
}

This turns:

Do I know enough?

into a more inspectable question:

Are the required fields present?
Are critical fields supported?
Are the sources admissible?
Are they fresh enough?
Are conflicts resolved?
Can missing evidence be recovered?

The contract is application-specific.

Evidence adequate for casual brainstorming may be inadequate for a financial filing, medical recommendation, deployment approval, or legal conclusion.

Epistemic adequacy is relative to the claim, reference regime, and action.

5. Evidence sufficiency is not one number

It is tempting to create another scalar:

adequacy = 0.83

Sometimes a summary score is useful.

The primary artifact should remain typed.

Suppose an answer requires atomic requirements:

SUPPORTED
PARTIALLY_SUPPORTED
INSUFFICIENT
CONFLICTING
STALE
UNRESOLVED
NOT_APPLICABLE

A weighted coverage summary might be:

Imagine:

company identity       SUPPORTED
currency               SUPPORTED
reporting framework    SUPPORTED
auditor                 SUPPORTED
Q3 revenue value       INSUFFICIENT

Four of five fields are present.

The one missing field is the answer.

So answerability must preserve:

coverage
+
critical-field status
+
conflict status
+
freshness
+
source admissibility
+
recoverability.

Again:

Preserve before scoring.

6. A typed epistemic-adequacy record

Answerability is the question — can this be answered now? Epistemic adequacy is the property the record carries, and state is its field. The rest of the book uses epistemic_adequacy as the record key (Chapter 12 onward).

A useful record might look like:

epistemic_record = {
    "state": "INSUFFICIENT_EVIDENCE",
    "required_fields": {
        "company_identity": "SUPPORTED",
        "reporting_period": "SUPPORTED",
        "revenue_value": "INSUFFICIENT",
        "currency": "SUPPORTED",
    },
    "critical_missing": ["revenue_value"],
    "evidence_conflict": False,
    "freshness": "ADEQUATE",
    "source_quality": "ADEQUATE",
    "recoverability": "RECOVERABLE",
    "next_action": "RETRIEVE",
    "suggested_query": "Company A Q3 revenue filing",
}

The vocabulary should distinguish at least:

ANSWERABLE
PARTIALLY_ANSWERABLE
INSUFFICIENT_EVIDENCE
CONFLICTING_EVIDENCE
UNDERSPECIFIED_INPUT
STALE_REFERENCE
UNVERIFIABLE
CAPABILITY_LIMIT

These states are not interchangeable.

They map to different recovery actions.

7. Answer, recover, or abstain

A mature system should not jump directly from:

not enough evidence

to:

I don't know.

Often the missing information can be recovered.

A better router is:

    graph TD
    Q[QUESTION] --> AC[ANSWERABILITY CHECK]
    AC --> SUFF{SUFFICIENT?}
    SUFF -- yes --> ANS[ANSWER]
    SUFF -- no --> MISS[WHAT IS MISSING?]
    MISS --> EXT[recoverable external evidence?] --> RET[RETRIEVE / TOOL]
    MISS --> USR[underspecified user input?] --> ASK[ASK CLARIFICATION]
    MISS --> CON[conflicting sources?] --> VER[VERIFY / RECONCILE]
    MISS --> CAP[capability limitation?] --> DEF[DEFER / ESCALATE]
    MISS --> UNR[not recoverable now?] --> ABST[ABSTAIN]
    MISS --> POL[policy prohibits action?] --> REF[REFUSE]
  

This matters because over-abstention is also a reliability failure.

A system that says:

I cannot determine that

while an authoritative tool could answer in 100 milliseconds is not prudent.

It is incomplete.

Recent retrieval work increasingly makes sufficiency judgment an explicit controller: if the current evidence is inadequate, the system emits structured gaps and retrieves again rather than prematurely answering or stopping.[2]

Similarly, work on selective visual reasoning has shown that low-confidence cases can sometimes be converted from abstentions into correct answers by gathering additional evidence first.[3]

The purpose of abstention is not to stop work. It is to stop unjustified commitment.

8. Partial answers are often better than all-or-nothing abstention

Suppose the user asks:

What were Company A's Q2 and Q3 revenues?

and the evidence contains:

Q2 revenue = $43.8 million
Q3 revenue = not present

A binary system has two options:

ANSWER
ABSTAIN

Both are unnecessarily crude.

A claim-level system can respond:

Q2 revenue was $43.8 million.
The supplied evidence does not contain Q3 revenue.

This is selective generation at claim granularity.

Let the proposed answer contain claims:

c1 → ACCEPT
c2 → ACCEPT
c3 → OMIT / VERIFY
c4 → ABSTAIN

This preserves useful coverage without allowing one unsupported subclaim to contaminate the whole response.

It also fits the architecture from Chapter 3:

claim decomposition
→ evidence state
→ policy action

9. Selective prediction formalizes the trade-off

Classical selective prediction represents a predictor together with a selection function.

Let:

Then coverage is:

This immediately exposes the central trade-off:

answer more
→ coverage rises
→ error risk may rise

abstain more
→ coverage falls
→ accepted answers may become safer

This (f, g) pair, with coverage $E[g]$ and selective risk $E[\ell g]/E[g]$, is the classical noise-free selective-classification formulation; the trade-off is summarized by a risk-coverage curve.[10][4][5] The reject option itself goes back to Chow’s error–reject analysis.[11]

This is much better than evaluating abstention with one refusal rate.

The system should be asked:

How much useful work can you cover at a specified accepted-risk budget?

That is the same systems logic we used for Hallucination Energy in Chapter 6.

10. Always abstaining is not reliability

There is a trivial way to avoid making false claims:

answer nothing.

At zero coverage, selective risk is not a meaningful achievement.

The system has simply stopped providing utility.

This is the abstention analogue of Chapter 10’s safe-but-useless model.

A useful reliability objective therefore needs both:

accepted risk
+
coverage.

Benchmarks are starting to score this pair directly. AA-Omniscience grades 6,000 factual questions on an index that penalizes a wrong answer and rewards abstention when the model does not know, and it reports that the highest-accuracy models do not lead — they lose points by guessing rather than abstaining, and only a few models clear an index of zero at all.[12] The best hallucination rate and the best accuracy belong to different models. That is the coverage–risk trade-off, observed in the field.

A simple expected-loss view can include:

For a casual recommendation, unnecessary abstention may be more costly than a minor factual miss.

For an autonomous medical or financial action, the relationship may reverse.

So:

The optimal abstention policy is an application policy, not a universal personality trait of the model.

11. Two symmetric abstention failures

Evaluation must punish both directions.

Over-answering

The oracle says the information is inadequate, but the model answers definitively.

missing evidence
→ confident completion

This is the classic hallucination-adjacent failure.

Over-abstention

The oracle says the information is sufficient, but the model refuses or hedges unnecessarily.

authoritative evidence available
→ "I cannot determine this"

This destroys coverage and user trust.

A minimal answerability confusion table is:

But production systems have richer actions than this table.

The more complete evaluation should distinguish:

ANSWER
PARTIAL_ANSWER
RETRIEVE
ASK_CLARIFICATION
VERIFY
ABSTAIN
REFUSE

and evaluate whether the reason and next action match the oracle state.

12. An abstention should identify the missing information

A bare:

I don't know.

is often better than a hallucination.

It is still not the best possible system behavior.

If the system can identify what prevents an answer, it should expose that gap.

For example:

I cannot determine Q3 revenue from the supplied evidence.
The missing item is the Q3 filing or another authoritative source
containing the Q3 revenue value.

This gives the user or retrieval system something actionable.

Recent 2026 abstention work explicitly optimizes not only correct abstention but also post-abstention clarification that identifies the key missing information.[6]

That suggests a stronger output contract:

abstention = {
    "state": "INSUFFICIENT_EVIDENCE",
    "missing": ["q3_revenue_value"],
    "reason": "requested_period_not_present_in_sources",
    "recoverable": True,
    "next_action": "retrieve_q3_filing",
}

Now abstention becomes a structured control signal rather than a dead end.

13. Data uncertainty and model uncertainty require different routing

Consider two queries.

Case A — data uncertainty

Should we choose Option A or B?

but the user has not provided:

budget
risk tolerance
deadline

The problem is not that the model lacks intelligence.

The input is underspecified.

Correct action:

ASK

Case B — model uncertainty

The user provides all necessary information, but the task requires a specialized calculation the current model cannot perform reliably.

Correct action may be:

TOOL
DEFER
HUMAN REVIEW

not:

ask the user to restate the same facts.

Recent 2026 work on LLM self-awareness evaluates this distinction directly and finds that high answer accuracy does not imply reliable attribution of whether uncertainty comes from the data or the model.[1]

This gives us an important router:

The action should reveal the diagnosis.

14. False premises require correction, not guessing

Some questions are structurally unanswerable because their premise is false or unsupported.

Suppose the user asks:

When did Company A acquire Company B?

but authoritative evidence says:

Company A never acquired Company B.

The system should not answer:

March 2024

and it should not necessarily respond with a vague:

I don't know.

The correct response is premise correction:

The evidence does not support the premise that Company A acquired Company B.

This matters because abstention benchmarks include failures such as false premises, underspecification, unknown answers, and outdated information.[7]

The right response to each is different.

So answerability is not merely:

CAN ANSWER
CAN'T ANSWER

It is diagnosis of why a definitive answer is not licensed.

15. Stale knowledge is another abstention problem

A model may have known the answer during training.

That does not mean its answer is current.

For time-sensitive questions, the answerability contract needs:

as_of date
source freshness
model knowledge boundary
retrieval freshness

A system should distinguish:

I have no evidence

from:

I have evidence, but it is too old for this question.

A stale answer can be perfectly contained in stale evidence and still be operationally wrong.

Freshness therefore belongs inside epistemic adequacy rather than being treated as a cosmetic metadata field.

16. Contradictory evidence should often increase restraint

Suppose two high-quality sources disagree:

Source A:
Event occurred on March 12.

Source B:
Event occurred on March 14.

The system has evidence.

It does not have resolved evidence.

A naive retrieval metric might say:

retrieval successful

because relevant passages were found.

An epistemic-adequacy check should say:

CONFLICTING_EVIDENCE

and route to:

verify
surface disagreement
seek authoritative source
or answer with qualified uncertainty

This is why more retrieved text does not necessarily increase answerability.

Chapter 8 showed that more context can create more recombination opportunities.

Chapter 11 adds:

More evidence can also reveal that the answer is less settled than it first appeared.

17. Retrieval should stop when the evidence is sufficient

Abstention and retrieval create a second trade-off.

Too little retrieval:

answer from incomplete evidence

Too much retrieval:

latency
cost
noise
contradiction accumulation
context dilution

So iterative RAG needs a stopping criterion.

A useful controller asks after each retrieval step:

Are all critical requirements supported?
If not, what exact evidence gap remains?

Recent ACL 2026 work operationalizes this with a structured sufficiency-and-gap judge that decides whether current evidence supports answering and, if not, emits gap items that drive the next retrieval query.[2]

The first-principles architecture is:

    graph LR
    RET[retrieve] --> JUD[judge sufficiency]
    JUD --> ADEQ{adequate?}
    ADEQ -- yes --> ANS[answer]
    ADEQ -- no --> GAP[identify gap]
    GAP --> RET_SPEC[retrieve specifically for gap]
    RET_SPEC --> JUD
  

A router earns reliability by mapping measured epistemic state to the least misleading next action, including actions that do not answer.

This is far stronger than:

retrieve top 10 passages
→ hope they are enough.

18. Help-seeking can be calibrated too

Abstention does not have to mean human intervention for every uncertain case.

For agents, the action can be:

ask user
call tool
request demonstration
escalate to human

KnowNo demonstrated this idea in LLM-based robotic planning: uncertainty calibration can be used to decide when an agent should ask for help, with conformal-prediction machinery used to provide formal assurances while limiting unnecessary human intervention.[8]

The broader systems lesson is more important than the specific method:

A reliable agent should know when autonomy ends and help-seeking begins.

That boundary should be evaluated like any other operating point.

19. Reasoning ability does not guarantee abstention ability

It would be comforting to assume that stronger reasoning models naturally know when not to answer.

Current evidence argues against that assumption.

AbstentionBench evaluates 20 frontier LLMs across 20 datasets and six broad abstention scenarios. It reports that scale alone provides little improvement in abstention, and that reasoning fine-tuning degraded abstention performance by about 24% on average in the evaluated models.[7]

More recent ACL 2026 work starts from the same observed weakness and proposes abstention methods specifically for reasoning models.[9]

This should change how we think about capability.

better at solving hard questions

is not the same capability as:

better at recognizing when the question cannot currently be solved.

Indeed, optimization pressure toward always completing a reasoning trace may sometimes make non-answering less natural.

The behavioral conclusion is enough:

Abstention must be measured explicitly. Do not infer it from benchmark intelligence.

20. A minimum abstention benchmark

A serious system should test multiple causes of non-answerability.

Answerable controls

complete authoritative evidence
single unambiguous answer
fresh source
no contradiction

Expected:

ANSWER

Missing evidence

Remove one critical field.

Expected:

RETRIEVE
or
ABSTAIN WITH GAP

Underspecified input

Remove one user-owned parameter.

Expected:

ASK CLARIFICATION

Conflicting evidence

Provide two credible incompatible sources.

Expected:

VERIFY / QUALIFY

False premise

Ask a question whose presupposition is contradicted by evidence.

Expected:

CORRECT PREMISE

Stale evidence

Provide authoritative but outdated material for a time-sensitive query.

Expected:

REFRESH / ABSTAIN

Capability limitation

Provide sufficient input but require a capability the tested system lacks.

Expected:

TOOL / DEFER

Irreducible uncertainty

Construct a case where no available action can resolve the uncertainty.

Expected:

ABSTAIN / QUALIFY

These are different tests because they require different diagnoses.

21. Use counterfactual answerability pairs

Chapter 9 taught us to evaluate transformations.

We can do the same for abstention.

Start with an unanswerable case:

Q3 revenue?

Evidence:
Q1 and Q2 only.

Expected:

ABSTAIN / RETRIEVE

Now add exactly one decisive evidence item:

Q3 revenue was $47.3 million.

Expected:

ANSWER

Then reverse the experiment.

Start answerable.

Remove the one critical field.

Expected:

ANSWER
→
ABSTAIN / RETRIEVE

This gives us a direct answerability-sensitivity test.

A model that answers both cases is overconfident.

A model that abstains on both is overcautious.

A model that switches appropriately has learned the relation we care about.

22. Evaluate the reason for abstention, not just the string

A naive abstention detector may search for phrases such as:

I don't know
I cannot answer
insufficient information

That is not enough.

Consider:

I cannot answer because this is private information.

when the real problem is:

Q3 evidence is missing.

The model abstained, but for the wrong reason.

Likewise:

I need the user's risk tolerance.

is wrong if risk tolerance was already provided and the actual missing item is the budget.

So evaluate:

ABSTENTION DECISION
Did the system avoid unjustified assertion?

UNCERTAINTY ATTRIBUTION
Did it diagnose the correct source of insufficiency?

GAP IDENTIFICATION
Did it identify what is missing?

RECOVERY ACTION
Did it choose an appropriate next step?

That is much harder—and much more useful—than checking for a refusal phrase.

23. A production answerability router

The full pipeline now begins to look like:

    graph TD
    Q[USER QUERY] --> DEC[CLAIM / TASK DECOMPOSITION]
    DEC --> REF[REFERENCE & EVIDENCE ACQUISITION]
    REF --> EP[EPISTEMIC ADEQUACY]
    EP --> ANS[ANSWERABLE] --> GEN[generate / verify answer]
    EP --> PAR[PARTIAL] --> PARANS[answer supported subset]
    EP --> UG[USER GAP] --> ASK[ask clarification]
    EP --> EG[EVIDENCE GAP] --> RET[retrieve]
    EP --> CONF[CONFLICT] --> VER[verify / reconcile]
    EP --> STALE[STALE] --> REFRESH[refresh]
    EP --> CAP[CAPABILITY GAP] --> TOOL[tool / human]
    EP --> UNRES[UNRESOLVABLE] --> ABST[abstain]
    
    POL[POLICY RESTRICTION] --> REFUSE[refuse]
  

Notice where policy refusal sits.

It is not another evidence state.

The epistemic router asks:

Can we justify an answer?

Policy asks:

Even if we can, are we allowed to provide or act on it?

That separation becomes crucial in Chapter 12.

24. The reliability record now includes epistemic adequacy

The diagnostic record is expanding:

reliability_record = {
    "containment": {
        "state": "LOW_RISK",
    },
    "consistency": {
        "relation_fidelity": "PASS",
        "polarity_fidelity": "PASS",
    },
    "sensitivity": {
        "counterfactual_decision": "PASS",
    },
    "epistemic_adequacy": {
        "state": "INSUFFICIENT_EVIDENCE",
        "critical_missing": ["q3_revenue_value"],
        "recoverability": "RECOVERABLE",
        "next_action": "RETRIEVE",
    },
}

This example matters because the first three families can all look fine.

The generated answer may be:

semantically ordinary
structurally valid
context-sensitive

while still lacking the evidence required to assert the value.

Epistemic adequacy therefore adds a genuinely new axis.

25. What you should now be able to answer

After this chapter, you should be able to explain:

1. Why abstention is a reliability capability rather than automatically a model failure.
2. Why model confidence and evidence sufficiency are different quantities.
3. The difference between abstention, clarification, retrieval, deferral, and policy refusal.
4. Why always abstaining cannot be considered a successful reliability strategy.
5. How risk and coverage create an operating trade-off for selective prediction.
6. Why partial claim-level answers can dominate binary answer/refuse behavior.
7. How to distinguish data uncertainty from model uncertainty.
8. Why an abstention should ideally identify the missing information and recovery action.
9. How to construct paired answerable/unanswerable cases by adding or removing one decisive evidence item.
10. Why reasoning benchmark strength does not imply good abstention behavior.

26. Exercises

Exercise 1 — Build an answerability pair

Create two versions of the same question.

Version A contains all critical evidence.

Version B removes exactly one critical fact.

Specify the expected action for each and test whether a model switches appropriately.

Exercise 2 — Separate clarification from retrieval

Construct one query where the missing information belongs to the user and one where it belongs to an external database.

The correct actions should differ:

ASK
vs
RETRIEVE

Exercise 3 — Test false-premise handling

Create a question with an invalid premise and provide authoritative evidence that contradicts it.

Check whether the system:

guesses
abstains vaguely
or corrects the premise.

Exercise 4 — Plot risk against coverage

Given a set of confidence or adequacy scores and correctness labels, vary the answer threshold.

Plot:

coverage
vs
selective risk.

Identify the threshold that satisfies a hypothetical application risk budget.

Exercise 5 — Audit abstention quality

For 20 abstentions, label:

correct abstention?
correct uncertainty source?
correct missing information?
correct recovery action?

Compare simple abstention rate with this richer evaluation.

27. The deeper lesson

The language model’s natural behavior is continuation.

Reliability requires interruption.

Sometimes that interruption means:

retrieve first

Sometimes:

ask the user

Sometimes:

verify the disagreement

Sometimes:

use a tool

And sometimes:

do not answer.

The important capability is not hesitation.

It is correctly identifying the boundary of justified commitment.

A reliable system should not ask only, “Can the model produce an answer?” It should ask, “What evidence authorizes this answer, and what should happen if that evidence is missing?”

That question takes us directly into policy.

So far, the book has built measurements:

containment
consistency
sensitivity
epistemic adequacy

The next problem is what to do with them.

When one signal fails, should we retrieve?

When two disagree, should we review?

When evidence is sufficient but provenance is weak, should we answer with qualification?

When the model is uncertain but the database is authoritative, which signal wins?

Those are not measurement questions anymore.

They are policy questions.

Research roots

1. Jingyi Ren, Ante Wang, Yunghwei Lai, Xiaolong Wang, Linlu Gong, Weitao Li, Weizhi Ma and Yang Liu, “Beyond ‘I Don’t Know’: Evaluating LLM Self-Awareness in Discriminating Data and Model Uncertainty,” ACL 2026. Introduces UA-Bench with more than 3,500 questions across six datasets and shows that frontier models struggle to distinguish input/data uncertainty from model uncertainty even when answer accuracy is high. https://aclanthology.org/2026.acl-long.547/

2. Minghan Li, Junjie Zou, Xinxuan Lv, Chao Zhang and Guodong Zhou, “S2G-RAG: Structured Sufficiency and Gap Judging for Iterative Retrieval-Augmented QA,” ACL 2026. Uses an explicit sufficiency judge that either authorizes answering or emits structured evidence gaps to drive subsequent retrieval. https://aclanthology.org/2026.acl-long.1185/

3. Tejas Srinivasan, Jack Hessel, Tanmay Gupta, Bill Yuchen Lin, Yejin Choi, Jesse Thomason and Khyathi Chandu, “Selective ‘Selective Prediction’: Reducing Unnecessary Abstention in Vision-Language Reasoning,” Findings of ACL 2024. Shows that additional evidence gathering can recover a substantial fraction of low-confidence examples without increasing error in the tested systems. https://aclanthology.org/2024.findings-acl.767/

4. Yonatan Geifman and Ran El-Yaniv, “Selective Classification for Deep Neural Networks,” NeurIPS 2017. Formalizes selective classification for deep networks using a risk-coverage trade-off and rejection to meet user-specified risk targets. https://papers.neurips.cc/paper_files/paper/2017/hash/4a8423d5e91fda00bb7e46540e2b0cf1-Abstract.html

5. Yonatan Geifman and Ran El-Yaniv, “SelectiveNet: A Deep Neural Network with an Integrated Reject Option,” ICML 2019. Trains classification/regression and selection jointly and evaluates the resulting risk-coverage trade-off. https://proceedings.mlr.press/v97/geifman19a.html

6. Haotian Zhai, Jingcheng Liang and Dongyeop Kang, “Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RL,” Findings of ACL 2026. Trains explicit abstention together with clarification that identifies the missing information on unanswerable queries. https://aclanthology.org/2026.findings-acl.985/

7. Polina Kirichenko, Mark Ibrahim, Kamalika Chaudhuri and Samuel J. Bell, “AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions,” 2025. Evaluates 20 frontier models across 20 datasets and six abstention scenarios, reporting weak abstention capabilities, little benefit from scale, and degraded abstention after reasoning fine-tuning. https://arxiv.org/abs/2506.09038

8. Allen Z. Ren et al., “Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners,” CoRL 2023. Introduces KnowNo, using conformal-prediction machinery to decide when LLM-based robot planners should ask for help while controlling uncertainty. https://proceedings.mlr.press/v229/ren23a.html

9. Abinitha Gourabathina, Inkit Padhi, Manish Nagireddy, Subhajit Chaudhury and Prasanna Sattigeri, “Answering the Wrong Question: Reasoning Trace Inversion for Abstention in LLMs,” ACL 2026. Proposes query-misalignment-based abstention for reasoning models and reports improvements across four frontier LLMs and nine abstention QA datasets. https://aclanthology.org/2026.acl-long.608/

10. Ran El-Yaniv and Yair Wiener, “On the Foundations of Noise-free Selective Classification,” Journal of Machine Learning Research 11, 2010, pp. 1605–1641. Defines a selective classifier as a predictor–selector pair (f, g) and formalizes the coverage / selective-risk trade-off. https://www.jmlr.org/papers/v11/el-yaniv10a.html

11. C. K. Chow, “On Optimum Recognition Error and Reject Tradeoff,” IEEE Transactions on Information Theory 16(1), 1970, pp. 41–46. The original analysis of classification with a reject option. https://doi.org/10.1109/TIT.1970.1054406

12. Artificial Analysis, “AA-Omniscience: Evaluating Cross-Domain Knowledge Reliability in Large Language Models,” 2025; arXiv:2511.13029. Scores 6,000 factual questions on the AA-Omniscience Index (−100 to 100), which penalizes hallucinated answers and rewards abstention when uncertain; finds the highest-accuracy models do not lead because they guess rather than abstain. https://artificialanalysis.ai/evaluations/omniscience

Next: From Measurements to Policy

We now have enough measurements to create a new danger.

A system can expose:

containment          = PASS
consistency          = PASS
sensitivity          = FAIL
epistemic_adequacy   = PARTIAL
provenance           = VERIFIED

and still leave us with the most important question unanswered:

What should the system do?

A measurement does not authorize an action.

Chapter 12 turns the diagnostic record into explicit policy: commitment is separated from recovery route.

commitment = PERMIT | HOLD | DENY
next_action = NONE | REFINE | RETRIEVE | VERIFY | ASK | REVIEW

That is where reliability becomes executable policy.