Hallucination Energy

Chapter 4 ended with a rule:

A hallucination measurement is a sensor, not a verdict.

This chapter builds one such sensor completely.

It begins with a property we care about:

semantic containment

But the first lesson of Chapter 4 was the proxy gap. We cannot inspect semantic containment directly. We can inspect representations.

So the construction in this chapter has three layers:

    graph TD
    LP[LATENT PROPERTY: semantic containment] --> RA[representation assumption]
    RA --> PP[PROXY PROPERTY: embedding-subspace containment]
    PP --> GC[exact geometric calculation]
    GC --> M[MEASUREMENT: squared projection residual]
  

That distinction is the chapter’s most important caveat.

Hallucination Energy does not directly measure semantic containment. It measures embedding-subspace containment, which we test as a proxy for semantic containment.

Suppose a claim is generated relative to a set of evidence passages.

If the meaning of the claim is represented well by semantic directions already present in the evidence, then its embedding should be largely explainable by an evidence-derived subspace.

If the claim introduces representational content not present in that subspace, some component should remain unexplained.

That unexplained component is what we measure.

We call it Hallucination Energy.

The name is deliberately narrower than the problem.

Hallucination Energy does not attempt to answer:

Is this claim true in the world?

Does a particular source entail it?

Is the citation authentic?

Did the model reverse subject and object?

Is the source sufficiently reliable?

Should policy allow the claim to be published?

It asks one geometric question:

How much of the claim representation lies outside the retained subspace represented by its evidence embeddings?

That gives us something unusually useful: a measurement whose assumptions, implementation, successes, and failures can all be inspected.

1. Start with the measurement contract

Before the mathematics, write the contract.

hallucination_energy_contract = {
    "target_property": "semantic_containment",
    "observable_proxy": "embedding_subspace_containment",
    "measurement": "squared_projection_residual",
    "measurement_unit": "claim_vs_evidence_set",
    "reference": "supplied_or_retrieved_evidence_embeddings",
    "output": "energy_in_[0,1]",
    "score_direction": "higher_means_less_contained",
    "requires_external_evidence": True,
    "requires_generator_internals": False,
    "known_blind_spots": [
        "relation_or_role_inversion_inside_evidence_span",
        "polarity_change_inside_evidence_span",
        "cross_evidence_feature_recombination",
        "bad_or_false_evidence",
        "provenance_failure",
        "citation_misattribution",
        "policy_admissibility",
    ],
}

This immediately prevents several overclaims.

Hallucination Energy is not:

probability that the claim is false

probability that the model hallucinated

probability that a citation is valid

probability that policy should accept the answer

It is a geometric measurement of one proxy property.

That distinction becomes more important, not less, when the experiments go well.

2. The assumptions connecting geometry to semantics

Every proxy needs assumptions connecting what we can measure to what we actually care about.

For Hallucination Energy, the important assumptions are explicit.

A1 — Representation fidelity

Semantically important differences must produce useful differences in the embedding representation.

If an embedder maps opposite relations to effectively the same representation, no downstream projection rule can recover information that the representation erased.

A2 — Approximate linear compositionality

Evidence-supported claim content must be reasonably approximable by combinations of evidence-derived directions.

This does not mean natural-language meaning is literally linear.

It means a low-rank linear subspace is being used as a tractable local approximation.

A3 — Novelty correspondence

Unsupported semantic material should often introduce an out-of-subspace component.

This assumption is strongest for failures such as:

new entity
new event
new topic
unsupported extension
semantic drift

It is weaker for failures that reuse all of the same semantic ingredients but bind them incorrectly.

A4 — Rank adequacy

The retained rank must be high enough to preserve legitimate evidence directions and low enough to avoid making the subspace trivially permissive.

Rank is therefore part of the measurement contract.

A5 — Evidence adequacy

The evidence set itself must be the correct reference for the task.

A geometrically perfect measurement against the wrong evidence is still the wrong measurement.

These assumptions tell us how to read later failures.

When CaseHOLD collapses, for example, the result is not simply:

geometry is bad.

A better diagnosis is:

the representation-plus-linear-subspace proxy does not preserve enough of the structural binding needed for this failure class.

That is a much more useful scientific statement.

3. Why pairwise similarity is not enough

Suppose we have a claim embedding:

Which single passage is closest to the claim?

But evidence is often distributed.

One passage may establish the entity.

Another may establish the date.

A third may establish the event.

A claim may therefore be reasonably represented by several evidence directions even when no individual passage is a close paraphrase.

That suggests a different geometric object.

pairwise similarity:
claim ↔ nearest evidence vector

subspace containment:
claim ↔ span of evidence vectors

The second is the proxy Hallucination Energy measures.

There is a reason to prefer projection residual over simply taking another distance heuristic.

For an orthonormal subspace, orthogonal projection gives the minimum-L2 approximation to the claim among all vectors in that subspace.

If:

It is the distance to the best linear reconstruction available inside the retained evidence subspace.

4. Build the evidence subspace

Let the evidence matrix contain one evidence embedding per row:

This is a design assumption:

we treat embedding direction as the primary semantic signal and intentionally discard vector-magnitude differences.

That is natural for cosine-oriented sentence embedding models, but it is not representation-neutral.

Let the normalized claim satisfy:

Retain the first \(r\) right-singular vectors as columns of a basis matrix:

there is no universal rank that belongs to Hallucination Energy independent of the measurement regime.

The rank must be recorded with the result.

5. Rank is a capacity control, not a harmless hyperparameter

For a fixed evidence matrix and the nested SVD basis, increasing retained rank can only increase explained mass.

Therefore:

Rank too low

Legitimate evidence directions are discarded.

supported claim
→ missing retained direction
→ inflated energy
→ false rejection risk

Rank too high

The subspace becomes increasingly permissive.

unsupported claim
→ still representable by broad span
→ deflated energy
→ false acceptance risk

As retained rank approaches the embedding dimension, the metric becomes uninformative because almost every vector can be represented.

Conceptually we want:

r = min(configured_cap, available_rank)

and then evaluate a rank sweep on the actual domain.

Variance-retention rules are another option:

choose the smallest r explaining at least q% of evidence spectral mass

but that changes the measurement definition and must itself be calibrated.

The current book results should therefore not be read as proving r=8 or r=32 is universally optimal.

A rank sweep on the HaluEval summarization pipeline — rank held at 1, 2, 4, 8, 16, 32, 64, everything else frozen to the run in Section 16 — makes this concrete:

Two things stand out. From rank 4 to rank 64 the bundle barely moves — a spread of about 0.004 AUC, well inside the bootstrap interval. And the low-rank end is if anything slightly better, not worse, which runs against the intuition that a too-small subspace discards legitimate evidence directions. On this workload, with this 13-feature bundle, rank is not a lever. The historical r = 32 choice sits on the flat part of the curve.

This is a bundle result, not a single-scalar one, and it is one dataset with one embedder; the isolated-energy and cross-embedder sweeps are still owed.

6. Project the claim into the evidence subspace

The projection of the normalized claim onto the evidence basis is:

    graph LR
    CV[claim vector] --> PROJ[project onto evidence subspace]
    PROJ --> CONT[contained component]
    PROJ --> RES[orthogonal residual]
  

The residual is the part of the claim representation not explained by the retained evidence span, which is why it becomes the energy signal.

If the claim lies entirely in the retained evidence subspace:

The proxy hypothesis is therefore:

small residual
    → claim representation is largely contained by evidence geometry

large residual
    → claim representation contains substantial out-of-span mass

Again, the inference stops there.

small residual ↛ true
large residual ↛ false

7. Residual magnitude and residual energy are different quantities

One natural quantity is normalized residual magnitude:

First compute explained mass:

Hallucination Energy is the squared normalized projection residual.

Mathematically, in exact arithmetic:

8. The one-vector special case: Hallucination Energy is \(\sin^2\theta\)

The simplest case gives useful intuition.

Suppose the evidence consists of one unit vector \(\mathbf e\).

Then:

one evidence direction
→ squared angular deviation

many evidence directions
→ squared deviation from an evidence subspace

Cosine similarity measures alignment with one direction.

Hallucination Energy generalizes the orthogonal deviation to a retained multi-direction evidence representation.

9. A two-dimensional example

Suppose the evidence spans only the horizontal axis:

64% of squared claim magnitude explained by retained evidence direction
36% remains orthogonal

The real embedding space may have hundreds or thousands of dimensions.

The decomposition is the same.

10. A reference implementation needs to handle evidence state explicitly

The geometric calculation is short, but edge conditions matter.

A robust educational implementation should validate shape before row normalization and preserve the distinction between missing evidence and observed high residual.

import numpy as np


def hallucination_energy(claim_vec, evidence_vecs, rank_r=8):
    c = np.asarray(claim_vec, dtype=np.float32)
    E = np.asarray(evidence_vecs, dtype=np.float32)

    if c.ndim != 1 or c.size == 0:
        raise ValueError("claim_vec must be a non-empty 1D vector")

    if E.size == 0:
        return {
            "state": "NO_EVIDENCE",
            "energy": 1.0,      # operational convention
            "explained": 0.0,
            "effective_rank": 0,
        }

    if E.ndim == 1:
        E = E.reshape(1, -1)
    if E.ndim != 2:
        raise ValueError("evidence_vecs must be a 2D matrix")
    if E.shape[1] != c.shape[0]:
        raise ValueError("claim/evidence embedding dimensions differ")

    c = c / max(np.linalg.norm(c), 1e-12)

    norms = np.linalg.norm(E, axis=1, keepdims=True)
    norms = np.where(norms < 1e-12, 1.0, norms)
    E = E / norms

    _, singular_values, Vt = np.linalg.svd(E, full_matrices=False)

    r = min(rank_r, Vt.shape[0])
    basis = Vt[:r].T

    coordinates = basis.T @ c
    explained = float(coordinates @ coordinates)
    energy = float(np.clip(1.0 - explained, 0.0, 1.0))

    return {
        "state": "MEASURED",
        "energy": energy,
        "explained": explained,
        "effective_rank": int(np.sum(singular_values > 1e-6)),
    }

The current Certum implementation surrounds the same core projection with additional diagnostics including:

singular-value ratios
participation ratio
entropy rank
similarity margin
alignment to dominant spectral direction
sensitivity
robustness probes

The key architectural principle is:

Do not collapse NO_EVIDENCE and MAXIMAL_OBSERVED_RESIDUAL into one semantic state merely because both can be represented numerically by 1.0.

The score is only one field in the measurement record.

11. The geometry itself has design choices

Hallucination Energy is generator-agnostic in one useful sense: the generator need not expose logits or hidden states.

But the metric is not representation-agnostic.

Its behavior depends on:

embedding model
embedding version
normalization convention
evidence segmentation
evidence selection
evidence duplication
centering convention
rank policy
claim resolution

Uncentered geometry

The current implementation performs SVD directly on unit-normalized evidence vectors.

It does not subtract an evidence centroid first.

So the detector uses an uncentered linear subspace through the embedding origin.

That is a genuine design choice.

Alternatives include:

centered PCA / affine subspace

globally whitened embeddings

weighted SVD

local nonlinear manifolds

We have not established that the current uncentered convention is optimal.

A centered-versus-uncentered comparison is therefore an important follow-up control.

Redundant evidence

Evidence order does not matter to the span.

Evidence multiplicity can matter under truncated SVD.

Repeating similar evidence can increase spectral weight along one direction and alter which components survive the rank cutoff.

So production systems should consider:

deduplication
source weighting
clustering
spectral diagnostics

rather than assuming repeated evidence is geometrically neutral.

Spectral cutoff stability

If:

Small evidence perturbations may rotate the retained basis substantially.

A useful diagnostic is therefore a spectral-gap measure around the cutoff.

The current Certum diagnostics already preserve singular-value structure; a production policy can use that information to flag an unstable subspace rather than pretending every basis is equally trustworthy.

12. A linear subspace is intentionally more permissive than evidence semantics

This is the deepest geometric limitation in the chapter.

A linear span contains arbitrary linear combinations of its basis directions.

It is not the set of propositions licensed by the evidence.

If \(\mathbf v\) lies in a subspace, then:

More importantly, semantic features can be recombined.

Consider:

Evidence 1:
Alice founded Alpha.

Evidence 2:
Bob founded Beta.

A wrong claim is:

Alice founded Beta.

The evidence contains all of the topical ingredients:

Alice
Bob
Alpha
Beta
founding

but not the asserted binding:

Alice = founder of Beta

If the embedding representation does not preserve the binding strongly enough, the wrong recombination can remain inside the same evidence span.

We will call this feature-binding failure or cross-evidence recombination failure.

Relation inversion is one example:

A acquired B
        ↓
B acquired A

Other examples include:

polarity reversal
causal direction reversal
temporal order reversal
role substitution
scope inversion
legal relation substitution

This yields a crucial statement:

The evidence subspace is a geometric relaxation of support, not a semantic proof system.

That is not a hidden weakness.

It is the exact boundary the CaseHOLD result in Section 20 begins to expose and Chapter 8 explains in full. A reader who stops at this chapter should still leave knowing the ceiling is there: containment can be perfect while the proposition is wrong, and no threshold on this metric fixes that.

13. Computational cost: keep the evidence window local

For an evidence matrix with \(n\) passages and embedding dimension \(d\), economy SVD is cheap when \(n\) is modest relative to \(d\), which is common in claim-level verification.

When \(n\ll d\), the dominant cost is roughly on the order of:

This is an important deployment constraint.

Hallucination Energy is designed to operate over a selected evidence set, not over every chunk in a corpus simultaneously.

Retrieval or evidence construction should happen first.

The current general Certum experiment configuration caps evidence sentences at 64; other pipelines expose their own limits.

For very large evidence sets, alternatives include randomized/truncated SVD, Gram-matrix eigendecomposition, or hierarchical evidence compression.

Those are engineering optimizations.

They do not change the measurement contract, provided the resulting basis is defined and recorded consistently.

14. From score to auditable gate

Once a continuous score exists, the temptation is to choose a threshold and call the detector complete.

Certum separates the stages.

    graph LR
    C[claim + evidence] --> E[embedding]
    E --> HE[Hallucination Energy]
    HE --> T[calibration-defined threshold τ]
    T --> P[policy]
    P --> D[accept / review / reject]
  

The simplest containment rule reduces the claim-evidence relationship to one scalar and then lets a fixed threshold make the decision.

For the simplest energy-only containment rule:

Passing this gate does not certify truth, provenance, attribution, or relational correctness.

It means only that the configured containment condition passed.

And determinism is not itself the reliability claim.

Given fixed:

embedding backend + version
evidence texts
segmentation
rank
numerical implementation
threshold

the calculation and decision are repeatable.

The engineering advantage is that the rule is explicit, inspectable, replayable, and auditable.

Those properties are more important than the word deterministic by itself.

15. The first real application: summarization

A clean atomic claim–evidence pair is the easiest case.

Summarization is harder.

A source document may contain dozens of propositions.

A generated summary may contain several supported sentences and one unsupported phrase.

A single document embedding and a single summary embedding can hide that local structure.

Hallucination Energy therefore has the same granularity problem Chapter 4 described for NLI.

A subspace detector applied at the wrong granularity can fail even when the underlying geometry carries useful signal.

The HaluEval pipeline used sentence-level decomposition:

    graph TD
    SD[source document] --> SE[split into evidence sentences]
    SE --> EEM[embed evidence sentences]

    SUM[summary] --> SS[split into summary sentences]
    SS --> SEM[embed each summary sentence]
    SEM --> GEO[compute geometry against document evidence]
    GEO --> AGG[aggregate sentence-level features]
  

The archived run loaded 10,000 HaluEval summarization rows and produced a 10,000 × 20 feature table.

Those 20 features included similarity, coverage, energy aggregates, entailment aggregates, and structural features.

That detail matters because the phrase geometry-only model does not mean the single Hallucination Energy scalar.

It means a model using a bundle of geometry-derived features, including:

mean_sim_top1
min_sim_top1
mean_sim_margin
min_sim_margin
mean_coverage
min_coverage
max_energy
mean_energy
p90_energy
frac_above_threshold
min_energy
energy_gap
high_energy_count

This is a critical distinction.

The chapter can currently support the claim:

geometry-derived containment features carry useful signal.

It cannot yet support the stronger claim:

the single Hallucination Energy scalar alone achieved AUC 0.7120.

A single-scalar baseline still needs to be reported separately.

16. What the 10,000-sample run actually establishes

The 10,000-row summarization experiment reported:

The geometry-bundle bootstrap interval was:

Projection-based geometry contributes substantial discriminative information in a real source-relative generative task.

These numbers were recovered from the config-preserved Certum summarization run 20260216_225831. Its emitted config.json records the full parameter set, and re-running the modeling stage reproduces every value in the table above, including the Section 18 ablation.

What the run records

dataset            HaluEval summarization, N = 10,000 rows
embedding model    sentence-transformers/all-MiniLM-L6-v2
NLI model          MoritzLaurer/deberta-v3-base-mnli-fever-anli
geometry rank      r = 32
geometry top_k     1000
run seed           1337   (data pipeline / evidence construction)
modeling seed      42     (train/test split, cross-validation, bootstrap)
20 extracted features and their family definitions
70/30 stratified row split; 5-fold stratified CV
reported AUC values and bootstrap intervals

What the run does not establish

single-energy-scalar AUC (the table is a 13-feature geometry bundle, not the H scalar)
simple cosine / centroid baselines
rank-sensitivity curve
source-document-grouped, entity-disjoint, or cross-embedder generalization

The split is stratified at the row level, not grouped by source document. So these are results about row-level generalization on one dataset with one embedder at rank 32.

This is not a reason to discard the experiment.

It is a reason to tighten the next one.

17. A split can leak structure even when labels are stratified

The current evaluation utility uses a stratified 70/30 row split for held-out logistic and XGBoost evaluation and 5-fold stratified row-level cross-validation.

That protects class balance.

It does not guarantee source-document isolation.

If multiple rows derived from the same source document can appear in the dataset, then the stronger protocol is:

all rows from source X
→ same fold

rather than:

one row from source X → train
another row from source X → test

The current runner does not enforce grouped splitting.

Therefore source-grouped evaluation is a required follow-up control before treating the summarization AUC as publication-grade evidence of transfer.

This is exactly the kind of distinction Chapter 6 will formalize:

random row generalization
≠
source generalization
≠
domain generalization

18. Before adversarial testing, perform a sanity check on feature claims

The full summarization model included correlated geometry features.

Removing individual features produced:

full model                  0.7284
without energy_gap          0.7284
without high_energy_count   0.7271
without both                0.7271

The correct conclusion is narrow:

Those individual removals did not materially reduce performance while related geometric substitutes remained available.

This does not prove the removed features are useless.

It does not prove Hallucination Energy is indispensable.

A stronger follow-up needs:

single-scalar energy baseline
max/mean cosine baselines
centroid-distance baseline
grouped ablation of all energy-derived features
permutation importance
rank sweep
embedding-model sweep

We do not invent those numbers here.

The absence of those controls is part of the current experimental record.

That honesty is more valuable than filling a table with unsupported precision.

19. The adversarial stress test attacks the proxy directly

Natural summarization asks whether the proxy carries useful signal.

Adversarial mining asks something harsher:

Can we deliberately construct unsupported examples that still look contained to the sensor?

The hard_mined_v2 procedure is the first step toward that.

    graph TD
    C[claim] --> EMB[embed the claim]
    EMB --> CC[compute centroids for candidate evidence sets]
    CC --> SH[shortlist top K=16 candidates by centroid similarity]
    SH --> EX[exclude own evidence and leakage sources]
    EX --> SEL[select most similar remaining mismatched evidence set]
  

This is a semantic-neighbour hard negative. The selection criterion is centroid similarity, with a leakage guard against reusing the claim’s own source. It is not energy-aware: the procedure does not search for the mismatch that the detector scores as safest.

The Certum implementation contains a second mode, hardest_energy_mined, that does compute Hallucination Energy for each shortlisted mismatch and select the minimum. That is the genuinely metric-aware attack. The experiments reported in this chapter used hard_mined_v2; the run reports record rerank_by_energy: false. The energy-aware attack is one of the controls still owed to this measurement.

The mining budget matters either way.

A search over 16 candidates is not equivalent to a search over 1,600.

Adversarial evaluation should therefore report performance as a function of search budget \(K\), rather than treating hard negative as a single fixed difficulty level.

20. Separate distributional separation from the operating point

The hard-mined experiments reported:

*The original experiment logs called this TPR_eval. Because hallucination can be treated as the positive class in other evaluation conventions, this chapter uses supported acceptance rate to make the orientation explicit.

Let:

μ_s = mean energy of supported examples
μ_n = mean energy of hard negatives

and:

Wikipedia  +1.92
PubMed     +2.11
CaseHOLD   −0.40

These numbers answer a distributional question.

The threshold columns answer a policy operating-point question.

Those are different.

Wikipedia is the warning

The Wikipedia regime showed substantial mean separation:

supported mean      0.3714
hard-negative mean  0.6950

but at the strict threshold the supported acceptance rate was only:

0.067

while false acceptance was:

0.006

There is no contradiction.

A strict false-acceptance budget can push the threshold deep into an overlapping distribution tail and destroy supported coverage.

That gives us a crucial lesson:

Mean separation is not an operating point.

Chapter 6 will turn that sentence into a full evaluation methodology.

21. CaseHOLD reveals structural binding loss

CaseHOLD behaves differently.

The hard-negative mean is lower than the supported mean:

A simple relation inversion shows why this is possible:

Evidence:
Company A acquired Company B.

Wrong claim:
Company B acquired Company A.

Both contain:

Company A
Company B
acquisition
same event domain

What differs is the binding:

A = buyer
B = target

versus:

B = buyer
A = target

The problem is therefore broader than topic overlap.

The missing information can be structural binding:

which entity has which role
which event precedes which event
which proposition is negated
which direction the cause runs
which legal relation applies to which party

If those distinctions do not produce sufficiently different representation geometry, both correct and incorrect candidates can be in-span.

The detector is not malfunctioning.

It is answering the question it was designed to answer:

Is this representation contained by the evidence subspace?

It was never a relation theorem prover.

This gives us a useful relationship with entailment.

As a design expectation:

strong explicit entailment
    should often imply low Hallucination Energy

if the embedder represents the relation well.

But:

low Hallucination Energy
    does NOT imply entailment

CaseHOLD and relation inversion are the counterexamples.

That is why geometry and entailment can carry complementary information.

22. We looked for an easy second scalar and did not find one

Once a useful scalar fails in an identifiable regime, the natural instinct is to add another scalar.

We tested variants including:

participation ratio
projection-related ratios
leave-one-out sensitivity
similarity margins
adaptive monotone combinations
gap-width tuning

The hope was that one orthogonal scalar would recover the missing separation under hard adversarial overlap.

It did not reliably do so.

That negative result matters because it distinguishes two very different problems:

ESTIMATION ERROR
right representation, noisy threshold / imperfect calibration

REPRESENTATION ERROR
sensor never encoded the distinction required by the failure

Calibration can improve the first.

It cannot manufacture a missing relational axis for the second.

When the proxy fails structurally, more calibration cannot repair information the sensor never observed.

That is one of the central principles of this book.

23. Weighted and nonlinear variants are possible—but they change the sensor

The current truncated basis treats each retained singular direction as a member of the subspace without weighting projection mass by singular value.

One alternative would define:

But it is no longer the same measurement.

Likewise:

centered affine subspaces
convex hulls
cones
kernel subspaces
local nonlinear manifolds
relation-aware graph embeddings

may address different geometric weaknesses.

The correct response to a structural limit is not to quietly mutate the metric until it passes one benchmark.

It is to name the new proxy and test it separately.

24. What Hallucination Energy measures well

The experiments support a bounded interpretation.

Hallucination Energy is most useful when unsupported content tends to introduce representational directions not already captured by the evidence set.

Examples include:

new entity
new event
new topic
unsupported contextual extension
semantic drift away from source
summary sentence poorly represented by source material

The metric also has practical properties.

Generator independence

It requires no generator hidden states or token probabilities.

Explicit reference

The score is defined against a concrete evidence set.

Continuous output

It can be ranked, calibrated, inspected near policy boundaries, and combined with independent diagnostics.

Interpretable geometry

A high value has a precise mathematical meaning: a large squared component remains outside the retained evidence subspace.

Cheap local computation

Once embeddings exist, the core measurement is linear algebra over a bounded evidence window rather than repeated generation.

Those are real engineering advantages.

25. What Hallucination Energy does not measure

The boundary is equally important.

Truth

False evidence can geometrically contain a false claim perfectly.

Directional entailment

A claim can remain in-span while reversing a relationship.

Structural binding

The right entities and relation words can be recombined into the wrong proposition.

Attribution

Low energy does not identify the precise supporting passage.

Provenance

Evidence may be circular, copied, stale, or fabricated.

Source reliability

The metric does not know whether a source is suitable for a claim.

Runtime state

If the model says it ran a tool it did not run, the correct detector is the runtime trace.

Policy admissibility

The metric does not know whether an action requires one source, two independent sources, a primary source, or human approval.

The clean formulation remains:

Hallucination Energy is a containment sensor. It is not a truth oracle.

26. Anti-patterns for this metric

A narrow sensor becomes dangerous when its contract is forgotten.

Do not use Hallucination Energy as:

A truth probability

H = 0.08

does not mean:

92% probability of truth

A cross-embedder universal scale

An energy threshold calibrated under one embedding model is not automatically meaningful under another.

A retrieval-independent score

Changing the evidence set changes the reference geometry.

Retrieval drift is measurement drift.

A threshold without an error budget

energy < 0.4 → accept

is not defensible until the operating point has been calibrated against the costs of false acceptance and false rejection.

A replacement for relation checking

Low energy is not permission to skip NLI, structured relation checks, or authoritative verification where those failure modes matter.

27. Controls still owed to the measurement

The experiments in this chapter are sufficient to establish that the proxy is interesting.

They are not the end of the evaluation program.

The highest-value follow-up controls are:

We do not fabricate those results in order to make the chapter look complete.

They are exactly the experiments a mature evaluation program should run next. The Evidence Ledger appendix collects them, with every measured number in the book and its provenance, in one place.

28. The chapter’s actual result

We can now state the contribution without overstating it.

The latent property was:

semantic containment.

The proxy was:

embedding-subspace containment.

The measurement was:

The experiments show three things.

1. The proxy carries real signal

On the 10,000-row HaluEval summarization run, a geometry-derived feature bundle achieved roughly 0.71 AUC, and combinations with entailment and structural features reached roughly 0.73–0.75 depending on evaluation configuration.

2. The proxy survives semantic-neighbour factual negatives

Wikipedia/FEVEROUS-style and PubMed hard-mined experiments — using the nearest mismatched evidence set, not an energy-aware adversary — retained substantial mean energy separation.

3. The proxy has a structural ceiling

CaseHOLD shows an observed structural ceiling; binding and relation-inversion examples explain how such failures can occur, while systematic binding-adversary experiments remain to be run.

Those three facts belong together.

If we report only the first two, we sell a detector. If we report all three, we understand a measurement.

That is the scientific method this book will use.

Research roots

This chapter is primarily a reconstruction of the Hallucination Energy experiments and the current open-source Certum implementation. External datasets are experimental environments rather than universal definitions of hallucination.

1. Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie and Ji-Rong Wen, “HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models,” EMNLP 2023, pp. 6449–6464. Introduces HaluEval, including generated and human-annotated hallucination examples used across several task settings. https://aclanthology.org/2023.emnlp-main.397/

2. Rami Aly et al., “FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information,” NeurIPS Datasets and Benchmarks 2021. Provides 87,026 claims with Wikipedia sentence/table evidence and SUPPORTS, REFUTES, or NOT ENOUGH INFO labels. https://arxiv.org/abs/2106.05707

3. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen and Xinghua Lu, “PubMedQA: A Dataset for Biomedical Research Question Answering,” EMNLP-IJCNLP 2019. Provides biomedical questions with PubMed abstract context and yes/no/maybe answers. https://aclanthology.org/D19-1259/

4. Lucia Zheng, Neel Guha, Brandon R. Anderson, Peter Henderson and Daniel E. Ho, “When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset,” 2021. Introduces CaseHOLD, a dataset of more than 53,000 multiple-choice examples for identifying holdings of cited legal cases. https://arxiv.org/abs/2104.08671

5. Ernan Hughes, Certum, open-source implementation of projection-based claim–evidence geometry, calibration, adversarial negative construction, policy gating, and evaluation. https://github.com/ernanhughes/certum

6. Junjie Hu et al., “HARP: Hallucination Detection via Reasoning Subspace Projection,” 2025. Uses SVD-derived internal reasoning subspaces and hidden-state projection for hallucination detection; Hallucination Energy instead operates at the external claim–evidence interface. https://arxiv.org/abs/2509.11536

7. Supratik Sarkar and Swagatam Das, “Grounding the Ungrounded: A Spectral-Graph Framework for Quantifying Hallucinations in Multimodal LLMs,” 2025. Develops a distinct spectral-graph and multimodal-manifold formulation. https://arxiv.org/abs/2508.19366

Next: How to Evaluate a Hallucination Detector

We now have a real sensor.

It has:

a latent target
an observable proxy
a mathematical definition
an executable implementation
continuous outputs
real experimental signal
known representation choices
known blind spots
unresolved controls

That is enough to begin evaluation properly.

And the numbers in this chapter have already shown why evaluation cannot collapse to one AUC.

A detector can have:

clear mean separation
and poor supported coverage at a strict FAR

or:

useful AUC on natural generation
and collapse under adversarial overlap

or:

stable behavior in one domain
and inverted ordering in another

So Chapter 6 asks the next first-principles question:

What does it mean for a hallucination detector to be good?

We will separate ranking from calibration, precision from recall, false acceptance from false rejection, benchmark difficulty from detector quality, row-level generalization from source-level generalization, and in-domain success from genuine transfer.

Only then can a measurement become a defensible policy instrument.