✨ TINY CRITICS: Lightweight Reasoning Checks for Large AI Systems

🥹 0. TL;DR

Large language models write fluent explanations even when they’re wrong. Verifying their reasoning usually requires another LLM slow, expensive, and circular.

We needed something different:

A miniature reasoning critic <50 KB trained on synthetic reasoning mistakes, able to instantly detect broken reasoning in much larger models.

The Tiny Critic:

• trains on GSM8K-style reasoning traces generated by DeepSeek or Mistral
• uses FrontierLens, and Visual Policy Maps (VPMs) to convert reasoning into canonical numerical features
• is just a logistic regression with ~30 parameters
• runs in microseconds
• plugs into any agent
• dramatically improves InitAgent, R1-Loops, and research-planning stability

This post tells the full story how we built it, why it works, and what we learned about the shape of reasoning.

🔑 Key Insights

1. Reasoning has geometry In Stephanie, good vs bad reasoning traces form distinct shapes in frontier space (FrontierLens).
2. Tiny models can ride that geometry A <50 KB critic over frontier features can reliably flag suspicious reasoning without another LLM call.
3. Shape beats raw scores Stability, mid-step dips, and frontier band occupancy often predict success better than any single scalar confidence.
4. Metrics are just signals The critic is metric-agnostic; it only cares whether a column helps separate good vs bad traces.
5. Ubiquity unlocks transparency Because the critic is so small, we can run it everywhere (agents, dashboards, browser) and make reasoning quality a first-class signal.

🐜 1. Why a Tiny Critic?

LLMs do not “know” when their reasoning is incoherent.

Across research agents, coding agents, and planning pipelines we saw the same pattern:

• Answers looked correct
• Reasoning felt plausible
• But the logic chain had breaks, jumps, or circular references

The only reliable way to catch it was to ask another LLM to double-check the first one.

That’s slow. That’s expensive. And the second model is trained on the same data regime and objectives, so it often repeats the same mistakes.

So instead of asking “Can we build a better thinker?”, we asked a narrower question:

🤏 Reliably flagging suspicious reasoning patterns?

Not: smarter than the base model. Just: good enough to say “this run looks shaky”.

The hypothesis is simple:

If we compress a reasoning trace into the right metrics (stability, mid-step dips, frontier coverage, entropy, etc.), then reasoning errors will cluster in a distinct region of that space.

If that’s true, we don’t need another huge model to detect them. We just need a small classifier sitting on top of those features.

🤷 Why make it tiny at all?

We could have used another transformer head, or a big MLP. We deliberately didn’t, for three reasons:

1. Independence The critic should be structurally different from the LLM it is watching. A tiny linear model over Frontier features gives us a very different failure mode than a next-token predictor.

2. Cost and ubiquity We want this to run:

   • on every reasoning trace,
   • inside long-running agents,
   • even in a browser or dashboard. That means: no GPUs, no model server, just w·x + b and a sigmoid.

3. Interpretability With a logistic regression we can:

   • inspect coefficients directly,
   • line them up with metric names,
   • ask “what frontier features does the critic actually care about?”. That’s crucial when you’re trying to debug both the critic and the system it supervises.

These constraints are what push us toward a Tiny Critic instead of “yet another big model”. The rest of the pipeline FrontierLens, FrontierIntelligence, SVM validation exists to make sure that tiny model has a clean, geometric view of reasoning quality to sit on top of.

👶 Can a tiny model reliably flag flawed reasoning?

We’re not trying to out-reason a 70B-parameter model.

The question is narrower:

Given a good feature space (stability, mid-step dips, frontier coverage, entropy, etc.), can a tiny model reliably tell “this trace looks trustworthy” from “this trace looks shaky”?

What we found is that reasoning errors aren’t random; they leave structural fingerprints:

• unstable score trajectories,
• “mid-step dips” where the chain loses the plot,
• high-entropy branches that never converge,
• FrontierLens bands where failed runs cluster in the wrong region.

Once those patterns are exposed as numbers, a small linear model does just fine at spotting them.
We don’t claim it thinks better than the base LLM only that it can cheaply recognize when the LLM’s reasoning looks like past failures.

✏️ Design constraints: what “tiny” actually means

From the start we imposed three hard constraints:

1. Size cap: The entire critic (scaler + weights + metadata) must stay under 50 KB. Concretely, that means:

   • a fixed, compact feature vector (tens of metrics, not thousands),
   • a single StandardScaler + LogisticRegression head,
   • no deep nets, no heavy embeddings, no extra model server.

2. Speed cap: Scoring one reasoning episode must be “JavaScript fast”:

   • a dot product + sigmoid,
   • no GPU,
   • cheap enough to run on every chain of thought without thinking about cost.

3. Composability: The critic must sit on top of existing signals, not replace them:

   • it consumes metrics from HRM, SICQL, EBT, Tiny, etc.,
   • it never touches the underlying LLM weights,
   • it can be dropped into any Stephanie pipeline as just another scorer.

Those constraints are what make the later “browser-native critic” possible. The model is small because the representation is doing the heavy lifting: FrontierLens, and our frontier-feature stack compress entire reasoning traces into a few dozen numbers.

The rest of this post is about how we turn full reasoning traces into a geometric space and then ride that geometry with a tiny model. The key tool there is FrontierLens, which we’ll meet in Section 4.

🤬 The Inner Critic: Giving AI the Ability to Question Its Own Reasoning

We all have that voice in our head the one that whispers “Wait, does that really make sense?” when we’re about to make a questionable decision. It’s our internal critic: that nagging, sometimes annoying, but ultimately invaluable part of our cognition that questions our assumptions, checks our reasoning, and steers us away from obvious mistakes.

For humans, this inner critic develops through experience. We learn from our errors, gradually building intuition about what “feels right” versus what’s likely flawed. But what if AI systems could develop something similar not another massive language model spouting opinions, but a genuine internal critic that operates quietly in the background, instantly flagging reasoning errors before they cascade into bad decisions?

That’s exactly what we’ve built: The Tiny Critic.

An alternate approach

Right now, when we want to verify if an AI’s reasoning is sound, we typically do something rather circular: we ask another large language model to critique the first one. It’s like having two people in a room arguing about which of them is more logical neither has an objective standard, and both might be equally wrong.

This approach is slow, expensive, and often fails to catch subtle reasoning flaws. Worse, the critic model frequently repeats the same mistakes as the model it’s evaluating. It’s a hall of mirrors with no exit.

Building a True Internal Critic

Instead of this circular critique pattern, we took inspiration from how humans develop reasoning intuition. We built a system that:

1. Learns from examples - Just as humans learn from experience, we trained our critic on thousands of reasoning traces, carefully labeled as “good” or “bad” reasoning.

2. Creates an efficient frontier - Drawing from financial portfolio theory, we defined a “frontier” between clearly bad and clearly good reasoning. This isn’t a simple binary threshold, but a dynamic band where the most interesting reasoning lives the zone where humans (and AIs) make their most subtle errors.

3. Operates with surgical precision - Unlike another LLM, our critic is tiny (less than 50KB) small enough to run in CPU cache, making decisions in microseconds rather than seconds.

4. Works silently in the background - Like your own internal monologue, it doesn’t generate text or explanations; it simply flags when reasoning starts to go off the rails.

How It Actually Works

Imagine watching a reasoning process unfold step by step. Traditional approaches might only check the final answer. Our Tiny Critic analyzes the entire trajectory of reasoning.

Using our Frontier Lens technology, we convert each reasoning step into a structured set of cognitive metrics measuring stability, consistency, logical flow, and more. These metrics form a geometric landscape where good reasoning follows predictable patterns.

The FrontierLens identifies the “efficient frontier” in this landscape the boundary between sound reasoning and potential errors. When reasoning starts to drift toward the “bad” side of this frontier, our Tiny Critic raises a quiet alarm.

This isn’t magic it’s geometry. Good reasoning creates recognizable patterns in this metric space, just as good investment portfolios create recognizable patterns in risk-return space. The Tiny Critic has learned to recognize these patterns through exposure to thousands of examples.

Simple utility

The power of this approach lies in its simplicity and efficiency. While another LLM might take seconds to critique reasoning (if it catches errors at all), our Tiny Critic works instantly fast enough to integrate directly into the reasoning process itself.

This means:

• Early error detection: Catching flawed reasoning before it compounds
• Resource efficiency: Running continuously without slowing down the system
• Objective standards: Using geometric patterns rather than subjective opinions
• Self-improvement: The system can learn from its own mistakes over time

Most importantly, it creates what we’ve been missing: an AI with the ability to question its own reasoning before presenting conclusions. Not with verbose explanations, but with the quiet confidence of a well-trained internal critic exactly what we humans rely on when making important decisions.

This isn’t just another verification tool. It’s the foundation for AI systems that can truly think for themselves questioning assumptions, recognizing flawed logic, and steering toward more reliable conclusions. In other words, it’s the first step toward AI that doesn’t just generate answers, but understands them.

    flowchart TD
    subgraph "🌊 Input Sources"
        A["🧠 Reasoning Traces<br/>DeepSeek/Mistral CoT"] 
        B["📊 Scorables<br/>Atomic Knowledge Objects"]
        C["📈 Metrics<br/>Step-Level Features"]
        D["🎯 HRM/SICQL Scores<br/>Quality Signals"]
        
        A --> B
        B --> C
        C --> D
    end

    subgraph "🔬 Frontier Analysis Engine"
        E["⚡ FrontierLens.from_rows()"]
        F["📋 Frontier Report<br/>Global + Regional Stats"]
        G["🎛️ Episode Features<br/>3M + 3 Vector"]
        H["🖼️ VPM Array<br/>Visual Policy Map"]
        I["🎚️ Dynamic Band Calc<br/>frontier_low/high"]
        J["🏆 Quality Score<br/>0.0-1.0"]
        
        C --> E
        D --> E
        E --> F
        E --> G
        E --> H
        F --> I
        I --> J
    end

    subgraph "🤖 Tiny Critic Core"
        K["🎯 Feature Vector Input"]
        L["🧠 Logistic Regression<br/>&lt;50KB Model"]
        M["💎 Reasoning Quality Score"]
        N["⚡ Decision Engine"]
        O["✅ Good Reasoning<br/>Clear & Logical"]
        P["⚠️ Risky Reasoning<br/>Needs Verification"]
        Q["❌ Bad Reasoning<br/>Flawed & Unreliable"]
        
        G --> K
        K --> L
        L --> M
        M --> N
        N --> O
        N --> P
        N --> Q
    end

    subgraph "🔄 Self-Improving Feedback Loop"
        R["📊 Critic Performance<br/>AUC & Accuracy"]
        S["🎯 Frontier Intelligence<br/>Metric Analysis"]
        T["🎪 Metric Selection<br/>Feature Importance"]
        U["🚀 Model Promotion<br/>Continuous Learning"]
        
        M --> R
        R --> S
        S --> T
        S --> U
        T --> I
        U --> L
    end

    subgraph "⚡ Real-Time Applications"
        V["🔄 Reject & Regenerate<br/>Flawed Reasoning"]
        W["💬 Request Clarification<br/>Borderline Cases"]
        X["🎉 Accept & Proceed<br/>High Quality"]
        Y["✨ Improved Reasoning<br/>Enhanced Output"]
        
        Q --> V
        P --> W
        O --> X
        V --> Y
        W --> Y
        Y --> A
    end

    %% Styling with vibrant colors
    classDef input fill:#e3f2fd,stroke:#1976d2,stroke-width:2px,color:#1565c0;
    classDef frontier fill:#e8f5e8,stroke:#2e7d32,stroke-width:2px,color:#1b5e20;
    classDef critic fill:#fff3e0,stroke:#ef6c00,stroke-width:2px,color:#e65100;
    classDef feedback fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px,color:#4a148c;
    classDef application fill:#ffebee,stroke:#c62828,stroke-width:2px,color:#b71c1c;
    
    class A,B,C,D input;
    class E,F,G,H,I,J frontier;
    class K,L,M,N,O,P,Q critic;
    class R,S,T,U feedback;
    class V,W,X,Y application;

    %% Enhanced connections with emphasis
    linkStyle 0 stroke:#1976d2,stroke-width:2px;
    linkStyle 1 stroke:#1976d2,stroke-width:2px;
    linkStyle 2 stroke:#1976d2,stroke-width:2px;
    linkStyle 3 stroke:#2e7d32,stroke-width:2px;
    linkStyle 4 stroke:#2e7d32,stroke-width:2px;
    linkStyle 5 stroke:#2e7d32,stroke-width:2px;
    linkStyle 6 stroke:#2e7d32,stroke-width:2px;
    linkStyle 7 stroke:#2e7d32,stroke-width:2px;
    linkStyle 8 stroke:#ef6c00,stroke-width:2px;
    linkStyle 9 stroke:#ef6c00,stroke-width:2px;
    linkStyle 10 stroke:#ef6c00,stroke-width:2px;
    linkStyle 11 stroke:#ef6c00,stroke-width:2px;
    linkStyle 12 stroke:#ef6c00,stroke-width:2px;
    linkStyle 13 stroke:#ef6c00,stroke-width:2px;
    linkStyle 14 stroke:#7b1fa2,stroke-width:2px;
    linkStyle 15 stroke:#7b1fa2,stroke-width:2px;
    linkStyle 16 stroke:#7b1fa2,stroke-width:2px;
    linkStyle 17 stroke:#7b1fa2,stroke-width:2px;
    linkStyle 18 stroke:#c62828,stroke-width:2px;
    linkStyle 19 stroke:#c62828,stroke-width:2px;
    linkStyle 20 stroke:#c62828,stroke-width:2px;
    linkStyle 21 stroke:#c62828,stroke-width:2px;
  

🧪 2. Data & Experimental Setup

Before we talk about frontiers, lenses, and tiny models, we should be clear about what we actually trained on and how we turned raw reasoning into geometry.

At a high level:

We took a single, well-understood math benchmark, let a medium model make both good and bad attempts, and then studied the shape of those reasoning traces.

This post focuses on one concrete setting:

• Task: GSM8K-style grade-school math word problems
• Base model: deepseek-math:7b running with a CoT-style prompt
• Labels: “correct final answer” vs “incorrect final answer”
• Goal: see whether good vs bad reasoning form separable clusters in a metric space rich enough to be interesting, but small enough for a tiny classifier.

Later, we’ll generalize this to other domains; here we just wanted a clean, controlled geometry experiment.

📀 2.1 Synthetic reasoning traces (GSM8K → DeepSeek)

The CriticDataAgent handles data generation:

1. Load a slice of GSM8K from Hugging Face.

2. For each problem, build a step-by-step reasoning prompt for DeepSeek-Math.

3. Call the model and capture the full chain-of-thought + final answer.

4. Canonicalize the answer (strip formatting, normalize numbers) and compare to the GSM8K gold label.

5. Wrap everything into a Scorable object with metadata:

   • is_correct (our ground-truth label for the critic)
   • basic stats: reasoning_length, step_count, verification_present, etc.
   • placeholders for downstream geometry features (stability, mid-dip, entropy, …).

6. Split into three cohorts:

   • scorables_targeted → correct solutions (our “good” cohort)
   • scorables_baseline → incorrect solutions (our “bad” cohort)
   • scorables → union of both (used by later stages)

This gives us a paired view of the same task: the same distribution of problems, solved well and badly by the same model under the same prompt, which is exactly what we want for studying reasoning geometry rather than dataset quirks.

📐 2.2 Turning traces into geometry

Raw text isn’t directly useful for a tiny critic. The next step is to project each reasoning trace into a feature space that captures its trajectory.

Here the Metrics stack kicks in:

• For every scorable, we run Stephanie’s scorers over the reasoning trace:

  • HRM (Hierarchical Reasoning Model) scores
  • SICQL Q/V/policy signals
  • Tiny risk / faithfulness / coverage heads

• On top of those, we compute Frontier “shape” features over the trace:

  • stability – how smooth the scores are across steps
  • middle_dip – whether the trace collapses in the middle
  • std_dev / volatility – overall wobbliness
  • entropy, trend, mid_bad_ratio, frontier_util, etc.

The important design choice:

Metrics are just signals. We don’t assume any one of them is “truth.” To the critic, they’re simply coordinates in a feature space.

That means this whole pipeline is metric-agnostic: you can swap out HRM for a different reasoning model, add new risk heads, or invent new shape features, and the critic training loop stays the same.

👾 2.3 From scorables to a critic dataset

Once teh Scorable Processor has done its work, we have per-scorable feature rows plus labels. The CriticDatasetAgent converts that into a clean training set:

1. Traverse the runs directory for this experiment.

2. Load the FrontierLens / metric matrices and the targeted vs baseline cohort reports.

3. Build a dense feature matrix X and label vector y:

   • X[i] = feature vector for reasoning trace i
   • y[i] = 1 if is_correct, 0 otherwise

4. Canonicalize and lock the metric names (so we can line them up across runs and models).

5. Project down to the 38 selected frontier features chosen by the MetricFilter + FrontierIntelligence step.

6. Save everything as a compact critic.npz:

   • X – feature matrix
   • y – labels
   • metric_names – ordered list of features
   • groups – optional group IDs (e.g., problem IDs) for analysis and CV

In the pilot run we describe in this post, that produced a small but balanced dataset (equal numbers of correct and incorrect traces), just enough to test whether a frontier direction exists and whether a tiny model can lock onto it.

🪪 2.4 What this experiment is (and isn’t)

This blog post is deliberately narrow:

• One dataset (GSM8K)
• One base model (DeepSeek-Math 7B)
• One critic architecture (logistic regression on ~30–40 features)

Within that sandbox, we wanted to answer two questions:

1. Geometry: Do good vs bad reasoning traces separate cleanly in frontier space?

2. Tiny critic viability: Given that separation, can a sub-50KB model learn a useful boundary that:

   • generalizes to fresh GSM8K traces, and
   • produces downstream gains when used as a triage signal?

The rest of the post walks through how we answered those questions:

• FrontierLens A/B plots to visualize the geometry,
• SVM validation to sanity-check the separating direction,
• a trained Tiny Critic to operationalize it, and
• downstream experiments to see if it actually moves the needle.

A more formal notebook and paper will follow; this post is the conceptual and architectural tour of how the data, geometry, and tiny model fit together.

🔭 3. FrontierLens: Seeing the Shape of Reasoning

The novel part here isn’t “we computed some metrics.”
It’s that we treat an entire reasoning trace as a point in a frontier space:

• each metric is a coordinate,
• the frontier metric defines a quality axis,
• the frontier band is the “interesting” region where reasoning can go either way.

FrontierLens gives us a compact picture of how whole cohorts of traces (lineups of wrong vs right reasoning) populate that space. Once you can see that separation by eye, training a tiny critic on top stops being speculative and starts being engineering.

🧐 3.1 Good vs Bad Reasoning Cohorts

Before training any critic, we wanted to answer a simpler question:

Do good and bad reasoning traces actually look different in frontier space?

To test this, we held the model and tasks fixed and only changed the labels:

• Baseline cohort → reasoning traces that failed (wrong final answer)
• Targeted cohort → reasoning traces that succeeded (correct final answer)

Both cohorts come from the same GSM8K-style problems, the same DeepSeek pipeline, and the same scoring stack. The only difference is whether the chain of thought ended in the right place.

For each cohort we:

1. extract all metrics (HRM, SICQL, Tiny, etc.),
2. build a score matrix (rows = episodes, cols = metrics), and
3. feed it through the FrontierLens.

That gives us a 3-panel “reasoning fingerprint” for failed vs successful traces:

FrontierLens failed reasoning (baseline cohort)	FrontierLens successful reasoning (targeted cohort)

Figure 3 Frontier geometry of wrong (left) vs right (right) reasoning traces, using HRM.coverage.attr.scm.knowledge.score01 as the frontier metric.

👓 How to read these plots

Each FrontierLens figure has three panels:

1. All metrics (left heatmap)

   • Rows = items (reasoning episodes) in this cohort
   • Columns = metrics (clarity, coverage, HRM scores, SICQL signals, etc.), normalized to [0,1]
   • The dashed vertical line marks the frontier metric here HRM.coverage.attr.scm.knowledge.score01

2. Frontier band (middle scatter)

   • X-axis = normalized frontier score
   • Y-axis = episode index
   • Dashed vertical lines = frontier band [low, high] (e.g. [0.25, 0.75])
   • Bright points = episodes inside the band; faded points = outside

3. Focused region (right heatmap)

   • Only the rows whose frontier score falls inside the band
   • Same normalization and metric ordering as the left panel
   • Title shows how many episodes land in-band (e.g. in-band rows: 39)

So for any cohort, the figure answers:

• What does the full metric space look like?
• Where do these episodes sit relative to the “good” band on the frontier metric?
• What internal metric pattern do the “okay-ish” episodes share?

🔀 What changes between failed and successful reasoning

When you compare the two cohorts, you’re seeing wrong vs right reasoning in the same metric space:

1. Band occupancy

   • In the failed cohort, many points in the middle scatter hug the low side of the band or never reach it.
   • In the successful cohort, the cloud shifts right: more episodes live inside the frontier band or above it, meaning stronger coverage/knowledge along the HRM frontier metric.

2. Focused-region coherence

   • For failed traces, the focused region (right heatmap) looks noisy and patchy: lots of nearly-dead columns, little structure across rows.

   • For successful traces, the focused region tightens up:

     • fewer “dead” metrics,
     • smoother vertical patterns,
     • clearer contrast between informative and noisy metrics. Visually, success looks like a cleaner, more compressible pattern in metric space.

3. Frontier metric behavior

   • In the failed cohort’s left panel, the frontier column spends more time in low or uncertain zones.
   • In the successful cohort, that same column is pulled into the band and high regions, and it co-moves more cleanly with other “good” metrics in the focused region.

In other words:

• Wrong reasoning → more mass outside the band, messy focused region, frontier metric misaligned with the rest of the score matrix.
• Right reasoning → more mass in-band or high, tidy focused region, and a frontier axis that behaves like a genuine “quality direction”.

These images are our first proof-of-concept:

Good and bad reasoning live in different parts of frontier space.

If we can see that separation by eye, then a small model sitting on top of these features should be able to learn the same boundary. The rest of the critic pipeline dataset construction, SVM validation, and the Tiny Critic itself is built on that observation.

🔄 How we generate these figures (for reproducibility)

For each cohort label ("failed" vs "successful"), we build a FrontierLens episode and render the figure:

vc = FrontierLens.from_matrix(
    episode_id=episode_id,
    scores=vpm,                 # rows = items, cols = metrics
    metric_names=metric_names,
    item_ids=item_ids,
    frontier_metric=frontier_metric,
    row_region_splits=self.row_region_splits,
    frontier_low=self.frontier_low,
    frontier_high=self.frontier_high,
    meta={"cohort": cohort_label},
)

img_path = self.out_dir / f"frontier_lens_{cohort_label or 'cohort'}.png"
render_frontier_lens_figure(vc, img_path)

FrontierLens.from_matrix(...) converts the metric matrix into a structured episode; render_frontier_lens_figure(...) produces the 3-panel plot used in Figure 3 for both failed and successful reasoning cohorts.

⛔ 3.2 A Single Trace, Seen Through Frontier Metrics

Here’s a toy GSM8K-style reasoning trace:

“To solve 3x + 5 = 20, subtract 5 from both sides: 3x = 15.
Then divide by 4 so x = 15/4. Close enough.”

The final answer is wrong (x should be 5), but the text sounds confident.

In frontier space this trace looks like:

• stability ≈ 0.3 scores wobble across steps,
• mid_step_dip ≈ 0.7 quality drops sharply in the middle of the chain,
• frontier_band_util ≈ 0.2 it spends little time in the “good” band,
• trend ≈ −0.2 overall quality drifts downward over time.

A similar trace that solves the equation correctly tends to have:

• higher stability,
• low mid-step dip,
• much higher frontier band utilisation,
• and a positive trend.

The Tiny Critic never reads the text. It just sees these fingerprints and learns that the first pattern usually ends badly and the second usually ends well.

🥣 4. Frontier Feature Set: What We Feed the Critic

One important caveat before we dive in: none of these metric names are sacred. To the critic they’re just columns of numbers that turned out to be useful at separating good from bad traces in this particular run.

Before we ask which signals the Tiny Critic leans on, we need to be clear about what we actually give it.

From all available metrics (HRM, Tiny, SICQL, shape features, etc.), the FrontierIntelligence + cohort pipeline selects a compact set of 38 frontier features. Each feature has:

• mean_target, mean_baseline average value on successful vs failed reasoning
• cohen_d, ks_stat, auc, effective_auc how well it separates good from bad
• direction whether “higher is better” (+1) or “lower is better” (-1) before correction
• is_core whether this feature is part of a small, hand-picked core set

At the source level, the 38 frontier features break down roughly as:

• Tiny metrics: 20
• HRM metrics: 12
• SICQL metrics: 2
• Global shape features: stability, middle_dip, std_dev, sparsity

That already tells a story: the frontier is built mostly out of Tiny + HRM signals plus a few shape features, with SICQL contributing a couple of targeted signals.

🪽 4.1 Top frontier features (by effective AUC)

If we sort selected_features.csv by effective_auc (AUC corrected for direction), the top frontier signals look like this:

A few quick takeaways:

1. HRM coverage really matters

   HRM.coverage.attr.scm.knowledge.score01 has the highest effective AUC (~0.85), and HRM.coverage.attr.ood_hat is not far behind. That’s strong evidence that epistemic coverage and OOD flags from HRM are genuinely predictive of whether a reasoning chain will end correctly.

2. Shape-of-reasoning features are powerful

   stability and std_dev are global shape features. High effective AUC here says: “the way the scores evolve over steps” (smooth vs jagged, stable vs volatile) is a strong discriminator between good and bad reasoning.

3. Tiny faithfulness / coverage are key axes

   The Tiny faithfulness & coverage heads (agreement, concept sparsity, coverage, uncertainty) all rank highly. When those heads are well trained, the critic inherits their discrimination almost for free.

This table is about feature quality, not yet about the critic’s decision. It answers:

“If all you knew was this single metric, how often could you guess good vs bad correctly?”

The Tiny Critic then takes these 38 frontier features together and learns an optimal linear boundary.

⚜️ 4.2 Tiny Critic coefficients: what the model actually uses

Once the Tiny Critic is trained, we can look directly at its logistic regression weights. After directionality correction, every feature is in a “higher = better” convention, so:

• positive coefficient → pushes toward “good reasoning”
• negative coefficient → pushes toward “suspicious reasoning”
• magnitude |coef| → how much that feature influences the decision

We extract, sort, and plot them:

from stephanie.components.critic.model.critic_model import CriticModel

model = CriticModel.load(DEFAULT_MODEL_PATH, DEFAULT_META_PATH)
coefs = model.clf.coef_[0]              # 1D array
names = model.meta.feature_names

# sort by absolute weight
idx = np.argsort(-np.abs(coefs))
top_k = 25

top_names = [names[i] for i in idx[:top_k]]
top_coefs = [coefs[i] for i in idx[:top_k]]

…and render them as a bar chart sorted by |coef|:

Tiny Critic metric importance. Bars show |coef| after directionality correction; the x-axis is absolute weight, and the y-axis lists the corresponding features.

In this particular run, the top weights are:

• Tiny.coverage.attr.vector.scm.uncertainty01
• HRM.coverage.attr.q_value
• HRM.coverage.score
• HRM.knowledge.attr.raw_score
• HRM.coverage.attr.ood_hat
• Tiny.faithfulness.attr.agree01
• Tiny.reasoning.attr.ood_hat01
• Tiny.coverage.attr.vector.scm.coverage.score01
• HRM.reasoning.attr.scm.uncertainty01
• Tiny.coverage.attr.aux3_p_good
• …plus shape features like trend and middle_dip further down

So the critic isn’t just copying the top AUC features it’s reweighting the whole frontier to find a boundary that works best when everything is combined.

You can read this plot almost like a diagnostic report on Stephanie’s scoring stack:

• Height (|coef|)

  • tall bar → the critic leans heavily on this feature
  • short bar → this feature barely moves the needle

• Sign (if you color by sign)

  • positive → “more of this looks like good reasoning”
  • negative → “more of this is a warning sign” (e.g., high uncertainty, strong mid-step dip, etc.)

Because we corrected directionality first, those signs have a clean human interpretation.

🎨 4.3 What this tells us about our scorers (and what to retrain)

This is where it stops being “just feature importance” and becomes system design feedback.

Patterns from the plot + AUC table together:

• Frontier metrics really dominate HRM coverage / q-values / OOD hats and Tiny coverage / uncertainty heads are clustered at the top. That’s exactly what we’d hope: the frontier definition itself is carrying real signal.

• Shape features are strongly predictive Features like trend and middle_dip show up with non-trivial weight. The critic cares about the shape of the reasoning trace, not just final scores.

• Some scorers underperform their intent Metrics that never appear in the top-k or always have near-zero weight are either:

  • noisy, or
  • not actually aligned with “good vs bad” on this dataset.

  Those are prime candidates for retraining or de-emphasizing in the next iteration.

This gives you a concrete loop:

1. Use effective AUC + coefficient plots to identify the most and least useful metrics.
2. Retrain / recalibrate upstream models whose signals matter most (e.g., HRM coverage, Tiny uncertainty).
3. Drop or downweight dead metrics.
4. Let FrontierIntelligence re-select frontier features on the updated stack.
5. Retrain the Tiny Critic and regenerate this same chart.

better scorers → cleaner frontier features → sharper critic → clearer feedback → better scorers …

And all of this is being driven by a model small enough to live inside a web page.

📺 4.4 Metric-agnostic by construction

One easy trap when you see a table of “Tiny.faithfulness…” and “HRM.coverage…” is to think:

“Ah, so the critic only works if you have these exact metrics.”

That’s not quite right.

From the critic’s point of view, metrics are just signals:

• a scalar that tends to be higher on good reasoning than bad (or vice-versa),
• plus enough consistency that we can estimate its direction and reliability.

The whole Frontier stack is built to be metric-agnostic:

• FrontierLens doesn’t care which score sits in column 37 it just sees “a frontier axis” plus bands.
• Shape features like stability, middle_dip, volatility are computed the same way no matter where the scores come from.
• The critic only ever sees a fixed-length feature vector; metric names matter to us, not to the model.

That’s why the metric filter report looks the way it does. For each candidate metric we log:

• how often it’s present,
• how well it separates good vs bad (AUC, KS),
• whether its direction is stable across cohorts,
• and any reason we drop it (e.g. “low support”, “direction flips”, “AUC ≈ 0.5”).

In other words:

We don’t start from a sacred list of “right” metrics.
We start from a large, messy pool of signals and let the FrontierIntelligence layer keep only the ones that behave.

If, a year from now, we swap in completely different upstream scorers new HRM, a different SICQL head, or even non-neural heuristics the Tiny Critic doesn’t care. As long as they produce repeatable signals, the same pipeline (filter → frontier features → critic training) still works.

🧱 5. The Foundation: Training the Tiny Critic

By this point we’ve seen that failed vs successful reasoning sit in different parts of frontier space. The next step is to turn that geometry into an actual model.

That’s the job of the CriticTrainer and CriticTrainerAgent.

At a high level:

1. critic_dataset builds a dataset of:

   • X: FrontierLens / HRM / SICQL features per episode
   • y: label 0 for failed reasoning, 1 for successful reasoning
   • groups: group IDs (e.g. GSM8K problem IDs) so we never leak the same problem across train/validation folds

2. critic_trainer turns that into a tiny but carefully trained critic:

   • directionality correction
   • feature locking
   • group-aware cross-validation and tuning
   • holdout evaluation + reports
   • a shadow pack for downstream agents

📝 5.1 From dataset to critic

The core trainer lives in CriticTrainer and is wrapped by CriticTrainerAgent so it can be used inside a Stephanie pipeline:

# inside CriticTrainerAgent.run(...)
result = self.trainer.train_from_dataset()
context["critic_stats"] = {
    "critic_model_path": str(result.model_path),
    "critic_meta_path": str(result.meta_path),
    "cv": result.cv_summary,
    "holdout": result.holdout_summary,
}

Under the hood, train_from_dataset() loads a NumPy .npz built by critic_dataset and calls train(X, y, metric_names, groups).

The train(...) method does six main things:

1. Audit the features It scans X for NaN/Inf values, logs which metrics are dirty, and optionally sanitizes them. This makes it very obvious when some metric is misbehaving without silently patching the data.

2. Directionality correction Some metrics are “higher is better” (e.g. coverage), others are “lower is better” (e.g. risk). The trainer applies a simple direction map:

   for i, name in enumerate(feature_names):
       d = self.directionality.get(name)
       if d == -1:
           Xc[:, i] = -Xc[:, i]
   

   After this pass, the critic can safely assume that larger values mean better reasoning along every axis.

3. Feature locking We don’t want the critic to chase a moving feature set on every run. The trainer can lock onto a stable subset of metrics, in priority order:

   1. names explicitly listed in cfg.lock_features_names

   2. names stored in the MetricStore meta for this run (metric_filter.kept)

   3. names listed in a lock_features text file

   4. otherwise:

      • if core_only=True, keep the first 8 “core” metrics
      • else, keep all metrics

   That gives you a robust way to say “these are the 30 features we trust; train only on them” while still letting FrontierIntelligence discover and store them automatically.

4. Group-aware split and CV The dataset is split into train vs holdout using GroupShuffleSplit when groups are available, so all traces from the same problem stay on one side of the split. For cross-validation, the trainer uses GroupKFold when possible, otherwise a stratified K-fold:

   cv, grp = _make_cv(groups, y)  # GroupKFold or StratifiedKFold
   

5. Tiny model + grid search

   The critic itself is deliberately simple:

   Pipeline(steps=[
       ("imputer", SimpleImputer(strategy="median")),
       ("scaler",  StandardScaler()),
       ("clf",     LogisticRegression(
           penalty="l2",
           solver="lbfgs",
           max_iter=2000,
           class_weight="balanced",
           random_state=42,
       )),
   ])
   

    flowchart LR
    A["📊 Frontier Feature Vector<br/>HRM / SICQL / Tiny"] 
        --> B["🔄 Directionality Correction<br/>(flip 'lower is better' features)"]
    
    B --> C["🔒 Feature Selection<br/>core + stored frontier features"]
    
    C --> D["⚡ Median Imputer"]
    
    D --> E["📏 StandardScaler<br/>mean/std normalization"]
    
    E --> F["🤖 LogisticRegression<br/>linear decision boundary"]
    
    F --> G["σ(w·x + b)<br/>probability of good reasoning"]
    
    G --> H["🎯 Critic Outputs<br/>critic_score, bad_reasoning_flag"]

    %% Styling with colors
    classDef dataNode fill:#e1f5fe,stroke:#01579b,stroke-width:2px,color:#01579b
    classDef processNode fill:#f3e5f5,stroke:#4a148c,stroke-width:2px,color:#4a148c
    classDef modelNode fill:#e8f5e8,stroke:#1b5e20,stroke-width:2px,color:#1b5e20
    classDef outputNode fill:#fff3e0,stroke:#e65100,stroke-width:2px,color:#e65100
    
    class A dataNode
    class B,C,D,E processNode
    class F,G modelNode
    class H outputNode
  

The trainer runs a small grid search over C:

param_grid = {"clf__C": [0.01, 0.1, 1.0, 10.0]}
gs = GridSearchCV(
    estimator=pipe,
    param_grid=param_grid,
    scoring="roc_auc",
    cv=cv,
    refit=True,
    error_score="raise",
)
gs.fit(Xtr, ytr, groups=grp)

Then it re-evaluates the chosen pipeline with a clean CV loop (AUC + accuracy) and on the group-aware holdout split, returning:

{
  "cv": { "auc_mean": ..., "auc_std": ..., "acc_mean": ..., "acc_std": ... },
  "holdout": { "auc": ..., "acc": ..., "n": ... }
}

6. Persist model + meta + reports

   Finally, the trainer:

   • saves the fitted pipeline with joblib.dump

   • writes a sidecar critic.features.txt so we can reconstruct the schema later

   • saves a JSON meta file with feature names, directionality, and metrics

   • calls generate_training_reports(...) to produce:

     • ROC curves
     • confusion matrices
     • coefficient plots and importance tables

   • writes a shadow pack (critic_shadow.npz) containing:

     • the holdout features/labels
     • the feature names actually used
     • any locked feature list from MetricStore

   The shadow pack is what lets CriticInferenceAgent and other tools replay exactly what the critic saw during training.

⛩️ 5.2 SVM frontier validation: checking the boundary

The logistic regression critic is the model we deploy, but we also wanted an independent sanity check that the frontier features really form a clean boundary between failed and successful reasoning.

For that, we add an optional SVM validation mode:

1. Take the same X, y frontier feature matrix used for training.

2. Train a linear SVM (LinearSVC) on a subset of high-signal frontier features.

3. Compute:

   • train AUC on the SVM decision values
   • hinge loss
   • margin statistics (mean, std, min, max)
   • support fraction (how many points sit on or inside the margin)

This produces a compact diagnostic like:

svm_result = {
    "enabled": True,
    "type": "LinearSVC",
    "n_samples": 100,
    "n_features": 102,
    "train_auc": 0.95,
    "hinge_loss": 0.49,
    "margin": {"mean": 0.54, "std": 0.42, "min": -0.31, "max": 1.68},
    "support_fraction": 0.95,
    "top_features": [...],
}

We then visualize the SVM decision values for failed vs successful episodes:

• Figure 4 SVM decision histogram Overlapping histograms of decision values (x-axis) for label 0 (failed) vs label 1 (successful). When the frontier is well-behaved, the successful histogram is clearly shifted to the right.

• Figure 5 SVM decision strip plot (optional) A scatter/strip plot of decision values along a single axis, colored by label. This makes the separating hyperplane visually obvious; successful traces cluster on one side, failed traces on the other.

The SVM is not the critic we don’t ship it in the pipeline. It’s a validation lens: if a simple linear SVM can’t find a clean separation, something is wrong with our frontier features or labels. When it can (high AUC, healthy margin stats, clear visual separation), it gives us extra confidence that the Tiny Critic’s decision boundary is grounded in real geometry, not just overfitting noise.

The punchline is simple:

• failed traces cluster on one side of the SVM decision axis,
• successful traces cluster on the other,
• the overlap is small, and the AUC hovers in the 0.9+ range.

That’s our independent proof that the frontier feature space really contains a usable quality direction. The Tiny Critic’s logistic regression isn’t hallucinating structure that isn’t there the separation is visible to any simple linear classifier we throw at it.

📢 6. Inference & Self-Promotion: Letting the Critic Compete With Itself

Training a Tiny Critic once is easy. Keeping it good as the rest of Stephanie changes is the hard part.

That’s what CriticInferenceAgent is for: it’s a self-promotion loop that lets new critic models quietly compete against the currently-promoted one on a fixed shadow dataset, and only swap them in when there is clear evidence of improvement.

At a high level:

1. Load a frozen shadow pack of features and labels.
2. Load the current critic and the candidate critic.
3. Project the shadow features into each model’s feature order.
4. Evaluate both models (AUROC, accuracy, calibration).
5. Decide whether to promote the candidate.
6. Export a teachpack of “interesting flips” for future training.

🥷 6.1 Shadow pack: a fixed battlefield

We keep a small “golden” evaluation set on disk:

@dataclass
class ShadowPack:
    X: np.ndarray             # frontier features
    y: np.ndarray             # 0/1 labels: bad vs good reasoning
    feature_names: List[str]  # column names
    groups: Optional[np.ndarray]
    meta: Dict[str, Any]

critic_shadow.npz is just a serialized ShadowPack.
It contains:

• the same frontier feature vectors used during training,
• labels marking good vs bad reasoning,
• optional group IDs (e.g., which GSM8K problem each example came from),
• and some metadata (dataset name, version hashes, etc.).

This file never changes during a run of experiments, so we can safely compare different critic versions on exactly the same data.

⛲ 6.2 Loading models and aligning features

The agent then loads:

• the current Tiny Critic from models/critic.joblib,
• a candidate from models/critic_candidate.joblib. :contentReference[oaicite:2]{index=2}

Each model may have been trained with a slightly different feature set. To make the comparison fair, we:

1. Recover each model’s expected feature names from its sidecar file <model>.features.txt, or fall back to the shadow pack’s names. :contentReference[oaicite:3]{index=3}
2. Reconcile that list with the model’s true width (n_features_in_), trimming or padding if necessary:

need_current = _reconcile_need_names(need_current, nfit_cur, tag="current")
need_candidate = _reconcile_need_names(need_candidate, nfit_cand, tag="candidate")

3. Project the shadow matrix into each model’s feature order, zero-filling any missing columns:

X_curr_proj = _project_to_names(X, have_names, need_current)
X_cand_proj = _project_to_names(X, have_names, need_candidate)
p_cur  = cur.predict_proba(X_curr_proj)[:, 1]
p_cand = cand.predict_proba(X_cand_proj)[:, 1]

This guarantees both models see the same examples, expressed in the way they were trained.

✔️ 6.3 Measuring competence and deciding promotion

On this shared battlefield, we compute AUROC and accuracy for each model:

from sklearn.metrics import accuracy_score, roc_auc_score

def _scores(y_true, y_prob):
    out = {}
    out["auroc"] = float(roc_auc_score(y_true, y_prob))
    y_hat = (y_prob >= 0.5).astype(int)
    out["accuracy"] = float(accuracy_score(y_true, y_hat))
    return out

We then make a simple, conservative decision:

• pick a comparison key (auroc or accuracy, configurable),
• promote the candidate only if it strictly beats the current model on that key (or if the current model has no valid score). :contentReference[oaicite:7]{index=7}

At the same time, we run a more detailed self-evaluation:

• self_evaluate(...) compares error patterns between current and candidate,
• update_competence_ema(...) maintains an exponential moving average of critic competence across runs, so we can track long-term drift. :contentReference[oaicite:8]{index=8}

The full result is written to runs/critic_inference_report.json a single JSON snapshot that includes:

• metrics for both models,
• the promotion decision,
• a small sample of feature names,
• and hashes of the feature layouts so we can spot accidental schema changes.

🎓 6.4 Teachpacks and promotion ledger

The most interesting examples are the flips:

• cases where the current critic gets it wrong but the candidate gets it right (bad→good), or vice versa.

We sort those flips by how much the probabilities changed and save the top few into a teachpack:

teachpack_file = teach_dir / f"teachpack_{ser.feature_fingerprint}.npz"
meta = teachpack_meta(fp_cand, need_candidate, calib=None)
export_teachpack(teachpack_file, X_cand_proj, need_candidate, y, p_cand, meta)

Later, the training pipeline can replay these teachpacks to:

• reinforce the candidate’s better behaviours,
• or study where it regressed.

If promotion goes ahead, we:

• overwrite critic.joblib with the candidate,
• refresh its sidecar feature list, and
• append an entry to a simple promotion ledger (promotion_ledger.jsonl) with fingerprints, metrics, and the decision. :contentReference[oaicite:11]{index=11}

That gives us a full audit trail of the critic’s evolution over time.

‼️ 6.5 Why this matters

This inference loop gives the Tiny Critic three important properties:

1. Safety: new versions don’t touch the live system until they’ve beaten the current one on a fixed, held-out shadow set.
2. Stability: we keep track of competence over time and can detect regressions or schema mismatches early.
3. Teachability: every promotion run generates targeted examples (teachpacks) that feed back into the training loop.

In other words, the critic doesn’t just run in real time it also tests and improves itself in the background, staying aligned with how Stephanie’s notion of “good reasoning” evolves.

💈 7. Self-Improver: When Does a New Critic Get Promoted?

Training a better Tiny Critic is useless if we accidentally promote a worse one.

Rather than doing something fancy, Stephanie uses a tiny, brutally simple guardrail: CriticSelfImproverAgent. Its job is:

1. Look at the freshly trained critic model + its eval metrics.
2. Compare it to the currently promoted critic.
3. Decide, using an explicit policy, whether it deserves promotion.
4. If yes, update a JSON registry; if not, record why it was rejected.

No magic, no hidden heuristics just a small, auditable rule set.

🎥 7.1 A 3-field model registry

At the end of training, we always have two files:

• models/critic.joblib the trained logistic regression,
• models/critic.meta.json its evaluation summary (AUROC, ECE, etc.).

The self-improver wraps these in a tiny ModelRecord:

@dataclass
class ModelRecord:
    model_path: str
    meta_path: str
    auroc: Optional[float] = None
    ece: Optional[float] = None
    pr_auc: Optional[float] = None
    accuracy: Optional[float] = None

and tracks the current champion in a JSON file:

class ModelRegistry:
    def __init__(self, path: str | Path = "models/critic_registry.json"):
        self.path = Path(path)
        self.path.parent.mkdir(parents=True, exist_ok=True)

    def load_current(self) -> Optional[ModelRecord]:
        ...

    def promote(self, rec: ModelRecord) -> None:
        payload = {"current": rec.__dict__}
        self.path.write_text(json.dumps(payload, indent=2))

That’s it: the “registry” is just critic_registry.json with one record called "current".

🤳 7.2 Promotion policy: dumb on purpose

The real logic lives in PromotionPolicy:

@dataclass
class PromotionPolicy:
    min_auroc: float = 0.70         # absolute floor
    max_ece: float = 0.20           # absolute ceiling
    require_improvement: bool = True
    auroc_margin: float = 0.01      # must beat current by > 1 pt
    ece_margin: float = -0.02       # and not get >2 pts worse in ECE

The ok(new, cur) method answers one question:

“Given the current critic and a new candidate, is it safe to promote the new one?”

Roughly:

1. Reject if the new model’s AUROC is below min_auroc.

2. Reject if its ECE (calibration error) is above max_ece.

3. If there is a current model and require_improvement is true:

   • new AUROC must be at least auroc_margin higher,
   • and ECE must not degrade beyond ece_margin.

If any check fails, we get a human-readable reason like:

AUROC gain 0.0040 < margin 0.0100 ECE delta +0.0300 > margin -0.0200

Those exact strings go into the run context and logs.

🎧 7.3 The CriticSelfImproverAgent in the pipeline

At the end of the critic training stage, the pipeline hands control to:

class CriticSelfImproverAgent(BaseAgent):
    async def run(self, context: Dict[str, Any]) -> Dict[str, Any]:
        # 0) make sure the trainer wrote model + meta
        # 1) parse meta → AUROC, ECE, PR-AUC, accuracy
        # 2) load current record from ModelRegistry
        # 3) ask PromotionPolicy.ok(new, current)
        # 4) optionally promote & update context

The agent:

1. Reads the meta file and pulls metrics from flexible locations (it uses deep_get / pick_auroc so it works with slightly different trainer formats).

2. Builds a ModelRecord for the new model.

3. Loads the current record (if any) from critic_registry.json.

4. Calls policy.ok(new_rec, cur).

5. Writes a summary into context["critic_self_improver"]:

   {
     "new": { "auroc": 0.81, "ece": 0.12, ... },
     "current": { "auroc": 0.79, "ece": 0.15, ... },
     "decision": "promote",
     "policy": { "min_auroc": 0.70, ... }
   }
   

6. If the policy says yes, it calls registry.promote(new_rec) which just overwrites the JSON.

No symlinks, no complex orchestration. Whatever points at critic_registry.json now implicitly points at the newly-promoted model.

🎶 7.4 Why separate this from training + inference?

Splitting training, inference, and improvement into separate agents gives you three nice properties:

• Swap-able policy You can tighten or relax the promotion thresholds without touching the trainer or inference code.

• Auditability Every run has a clear “promote vs reject (reason)” entry in both logs and context. If a critic regresses, you know exactly why it was allowed or blocked.

• Safety in motion As you retrain upstream scorers (Tiny, HRM, SICQL) or change the frontier feature set, the self-improver ensures the critic only moves forward when the overall evaluation story is better, not just when one metric jiggles up.

In other words: the Tiny Critic doesn’t just get trained once and frozen; it lives inside a tiny, explicit promotion contract that keeps it honest as the rest of Stephanie evolves.

📒 8. Metrics as Signals, Not Beliefs

Up to this point I’ve been talking about “HRM coverage”, “Tiny faithfulness”, “stability”, etc. in very semantic terms.

Under the hood, the Tiny Critic doesn’t care.

To the critic, each metric is just:

a column of numbers that sometimes helps separate good from bad reasoning.

That’s it. The meaning of the column is entirely a property of the upstream model that produced it. The critic pipeline is deliberately metric-independent: it will happily work with any set of signals, as long as they can be turned into a matrix.

🔞 8.1 Metric filter explanations

To make that explicit, every run produces a small report:

metric_filter_explain.csv

Each row is one metric and one decision:

• metric full metric name
• action "kept" or "dropped"
• reason why we dropped it (or why it survived)
• optional diagnostics: auc, cohen_d, is_constant, nan_fraction, etc.

A toy excerpt might look like:

This is the first guardrail layer:

• we don’t trust a metric just because it exists,
• we keep metrics that actually separate failed vs successful reasoning,
• we drop metrics that are constant, noisy, missing, or behave like random noise.

Crucially, this logic does not special-case HRM, Tiny, SICQL, or any other family. It only cares about the statistics of the column.

🧊 8.2 Metric-independent by design

This leads to a subtle but important design property:

The entire frontier + Tiny Critic pipeline is agnostic to the specific metrics.

• Today, the columns come from HRM, Tiny, SICQL, and frontier shape features.

• Tomorrow, they could come from:

  • a different HRM variant,
  • vision models over VPM tiles,
  • code-specific scorers,
  • or totally new research agents.

As long as you can:

1. arrange them into a matrix X of shape (episodes, metrics), and
2. supply labels y = {failed, successful},

the same pipeline will:

• evaluate single-metric quality (effective AUC, Cohen’s d, KS),
• explain which metrics were kept or dropped (metric_filter_explain.csv),
• learn a new frontier feature set,
• and train a fresh Tiny Critic on top.

From the critic’s perspective, metrics are coordinates in a space, not hard-coded beliefs about the world. The semantics only matter to us when we look back at:

• which axes it chose to trust, and
• what that says about the upstream models we should retrain or retire.

That’s exactly why this approach is reusable:

• swap the metrics,
• rerun the pipeline,
• read a new metric_filter_explain.csv,
• and you have a Tiny Critic tuned to a completely different domain with the same <50 KB model footprint.

📎 9. Downstream Impact: Does the Critic Actually Help?

So far, everything has been internal: metrics, frontiers, feature weights.

To test whether the Tiny Critic actually matters, we plug it into a tiny downstream task and ask a very practical question:

If I can only “spend” a small review budget, does the critic help me pick better examples than random or naive top-p?

9.1 Experimental setup

We start with a small GSM8K-style dataset:

• 48 reasoning episodes with ground-truth correctness labels
• A critic probability for each episode (higher = “looks like good reasoning”)
• A simple task accuracy function: given a set of indices, return the fraction that are actually correct

Then we evaluate three selection strategies over different budget levels (fraction of examples we’re allowed to keep):

results = compute_downstream_impact(
    y_true,             # 0/1 correctness
    critic_scores,      # TinyCritic probabilities
    accuracy_func,      # e.g. GSM8K accuracy
    budget_levels=[0.01, 0.05, 0.10, 0.20, 0.30, 0.50],
    n_simulations=100   # for baselines
)

For each budget we compare:

1. Critic selection take the top-k examples by critic score.
2. Random selection sample k examples uniformly at random (100 simulations; we report mean ± std).
3. Top-p selection a stand-in for “use the model’s own confidence”: we sample from the top half of examples as if we were trusting its logits.

The line plot you saw (Downstream Impact on Task Accuracy) is exactly this experiment.

📤 9.2 What the curves show

A few patterns are immediately visible:

• At very tiny budgets (1%) everyone is unstable: the critic curve starts at 0.0 (we happened to pick the wrong example), and random/top-p bounce around because there are so few samples. This is more of a sanity check than a useful regime.

• At 5-20% budgets, where the regime is realistic, the critic starts to shine:

  • 5% budget

    • Critic ≈ 0.50 accuracy
    • Random ≈ 0.47 (with huge variance)
    • Top-p ≈ 0.63
    • This is the “triage is barely starting” zone; the critic is competitive but noisy.

  • 10-20% budget

    • Critic: 0.75 → 0.78
    • Random: ≈ 0.49
    • Top-p: ≈ 0.60 0.63

    Here the critic is clearly doing useful work: +26 28 points over random, and +14 15 points over the top-p baseline.

• At larger budgets (30-50%), everyone converges:

  • Critic gently drifts down from ≈0.71 to ≈0.63
  • Random creeps up toward ≈0.51
  • Top-p hovers around ≈0.63

Once you’re reviewing half the corpus, selection becomes less important; the critic’s advantage shrinks but doesn’t reverse.

🧑 9.3 Bar-chart view (simplified report)

For the blog we also generate a simpler, 4-budget bar chart using a helper:

report = run_downstream_experiment(
    y_true=y_true,
    probs=critic_scores,
    accuracy_func=accuracy_func,
    budget_levels=[0.05, 0.10, 0.20, 0.50],
    output_dir="reports/downstream",
)

This produces both the figure and a markdown report:

Figure Random vs Critic selection for 5%, 10%, 20%, 50% budgets.

The table behind it looks like:

This smaller experiment is noisier, but it tells a consistent story:

• At tiny budgets (5%), critic-based triage is dramatically better than random (jump from 0.0 → 0.50 in this run).
• At 50% budget, the critic still provides a +21-point lift over random by picking better half-corpora.

💼 9.4 How to read this in practice

The key takeaway is not that we squeezed out a specific percentage, but that:

Once you step into a realistic review budget (5 20%), a tiny critic trained on frontier features can consistently pick better reasoning chains than either random or naive top-p selection.

In other words:

• If you’re a human reviewer with limited time,
• or another agent that can only afford to “trust” a small fraction of chains,

the Tiny Critic acts as a triage layer:

• push strong candidates to the front of the queue,
• quietly down-weight chains whose geometry looks wrong,
• make better use of whatever budget you have.

The important part is that this improvement is downstream: it’s not just a nicer ROC curve, it’s more correct answers for the same amount of work.

✊ 10. Why This Changes How We Think About Reasoning

For me, the interesting part of this project isn’t that we squeezed a bit more AUC out of a GSM8K corpus.

It’s that:

• Reasoning quality turns out to have a stable geometric signature.
• A tiny model is enough to read that signature.
• That tiny model is cheap enough to run everywhere.

Put those together and you get a different way of building systems:

• don’t just ask models for answers – always record their reasoning traces,
• embed those traces into frontier space,
• learn tiny critics that watch the geometry,
• treat critic scores as first-class signals for triage, safety, and self-improvement.

The long-term goal is simple:

any time an AI produces a chain of thought, it should also produce a tiny, honest score about how much that chain looks like past successes.

Once that’s cheap and ubiquitous, “reasoning-transparent AI” stops being a slogan and becomes just another layer in the stack.

🧩 11. Conclusion & Next Steps

The original question behind this project was small but sharp:

Can a tiny model, sitting on top of the right metrics, reliably tell when a much larger model’s reasoning looks wrong?

By the end of this experiment, we’ve answered that with a qualified but solid yes.

What we actually showed in this post:

• Reasoning has a geometry.
  On a single, well-controlled setup (GSM8K + DeepSeek-Math), failed vs successful reasoning traces do not overlap randomly. In FrontierLens space, they occupy measurably different regions, with cleaner patterns, stronger coverage, and more stable trajectories on the “good” side.

• Frontier features are enough for a tiny model to read that geometry.
  A simple logistic regression on ~30–40 frontier features no transformer, no embeddings can learn a boundary that:

  • generalizes to a held-out set,
  • passes an independent SVM sanity check, and
  • produces real downstream gains when used as a triage signal under realistic review budgets.

• Metrics are just signals, not beliefs.
  The critic doesn’t “believe” in HRM, Tiny, SICQL, or any particular score. It sees a matrix of numbers and learns which columns actually separate good from bad traces. That metric-agnostic design is what makes the whole approach reusable: change the scorers, rerun the pipeline, get a new critic.

• Tiny is a feature, not a gimmick.
  Keeping the critic under ~50 KB wasn’t a stunt it was a design constraint. It forced us to:

  • invest in the representation (FrontierLens, shape features, metric filtering),
  • make the model interpretable (weights line up with named metrics), and
  • make deployment trivial (dot product + sigmoid, even in a browser).

• Self-promotion and self-improvement are built in.
  The CriticInferenceAgent and CriticSelfImproverAgent treat each new critic as a candidate, not a replacement. New models must beat the current one on a fixed shadow set and obey a simple, explicit promotion policy. That gives us a tiny but real learning-from-learning loop around the critic itself.

Taken together, this gives us something we didn’t have before:

a portable, inspectable, self-improving reasoning critic that can sit beside any chain-of-thought system and quietly say,
“this looks like the kind of reasoning that usually ends well” or
“this looks worryingly similar to past failures.”

🚶 Where this goes next

This post stayed deliberately narrow:

• one dataset (GSM8K),
• one base model (DeepSeek-Math 7B),
• one critic architecture (logistic regression on frontier features).

The next steps are where it gets interesting:

1. New domains and modalities
   Swap GSM8K for code traces, research agents, multi-step planning, or even visual VPM tiles. The pipeline stays the same; only the metrics and labels change.

2. Multiple critics, different roles
   Train separate Tiny Critics for:

   • hallucination risk,
   • epistemic coverage,
   • safety policy violations,
   • or “this will waste a reviewer’s time.” They can run side-by-side, each watching a different facet of the same trace.

3. Deeper integration into Stephanie
   Use critic scores as:

   • gates in R1-style loops,
   • weights in policy selection,
   • signals for when to call heavy scorers (HRM, SICQL) vs cheap ones,
   • or triggers for saving / discarding traces in long-running “habitat” runs.

4. Browser-native and edge-native deployments
   Export the critic as JSON and let dashboards, IDEs, notebooks, and agents decorate reasoning with live quality badges no extra API calls, no heavy model server.

In that sense, this project is less about one tiny logistic regression and more about a pattern:

• record the reasoning,
• embed it into a shared frontier space,
• let a small, honest critic learn the geometry,
• and wire that signal back into how the system behaves.

If we keep doing that across more domains, more agents, more metrics the idea of “reasoning-transparent AI” stops being marketing language and becomes the default way we build and debug large systems.

This post is the first concrete slice of that vision: a small, working proof that geometric critics are both feasible and useful. The notebook and paper that follow will go deeper into the math, but the core message will stay the same:

You don’t always need a bigger model to get better reasoning.
Sometimes you just need a sharper lens, a cleaner space, and a tiny critic watching the frontier.

🐍 Appendix 1: An End-to-End Run: From Traces to Critic Decisions

To make all of this less abstract, here’s a single end-to-end run of the Tiny Critic pipeline, start to finish.

We’ll walk through one concrete experiment:

GSM8K-style math problems → DeepSeek-Math chain-of-thought → metric traces → FrontierLens → Tiny Critic → downstream selection.

🏆 1. Data: failed vs successful reasoning

We start with a batch of GSM8K questions and ask DeepSeek-Math 7B to solve them with full chain-of-thought.

From this batch we extract:

• 137 scorables (reasoning episodes)

• Each tagged as:

  • baseline → final answer wrong (failed reasoning)
  • targeted → final answer correct (successful reasoning)

In this run:

• Targeted scorables: 100
• Baseline scorables: 37
• Positive rate in the held-out evaluation set: 50% (24/48).

The idea is simple:

If the frontier geometry and critic are doing anything useful, the failed and successful episodes should look different in that space – and the critic should be able to tell them apart.

🙌 2. FrontierLens: good vs bad geometry

For this run we choose the frontier metric:

HRM.coverage.attr.scm.knowledge.score01

and a frontier band of [0.25, 0.75], with 4 row regions (early → late).

We build one FrontierLens episode per cohort:

frontier_metric = "HRM.coverage.attr.scm.knowledge.score01"

vc = FrontierLens.from_matrix(
    episode_id=episode_id,
    scores=vpm,                 # rows = items, cols = metrics
    metric_names=metric_names,
    item_ids=item_ids,
    frontier_metric=frontier_metric,
    row_region_splits=4,
    frontier_low=0.25,
    frontier_high=0.75,
    meta={"cohort": cohort_label},
)

render_frontier_lens_figure(vc, f"frontier_lens_{cohort_label}.png")

This produces the two 3-panel figures we showed earlier:

• frontier_lens_baseline.png – failed reasoning episodes
• frontier_lens_targeted.png – successful reasoning episodes

In this particular run:

• Frontier quality score

  • Targeted: ~0.555
  • Baseline: ~0.392
  • Δ(target − baseline): +0.163

• Frontier band occupancy (global frontier_frac)

  • Targeted: 0.39
  • Baseline: 0.32

You can see it visually:

• In the baseline plot, more episodes sit in the low region of the frontier metric; the focused (in-band) region looks noisy and uneven.
• In the targeted plot, more episodes move into the frontier band; the focused region is tighter and more structured.

That’s our first proof that:

“Good vs bad reasoning” is not random noise – it has a stable geometric signature in frontier space.

☑️ 3. Frontier feature vector + SVM sanity check

Next, we turn these metric matrices into frontier features:

• Per-metric means, std, frontier band fractions
• Global shape features: stability, mid-step dip, volatility, sparsity, entropy
• Frontier geometry summaries per region

For this run we end up with:

• 102 frontier features per episode (before any pruning)
• 137 samples (same as the scorable count)

To sanity-check that the frontier space is actually separable, we train a tiny linear SVM only for validation:

svm = LinearSVC()
svm.fit(X_frontier, y_labels)          # 137 × 102
svm_scores = svm.decision_function(X_frontier)
train_auc = roc_auc_score(y_labels, svm_scores)

The SVM report for this run:

• Model: LinearSVC
• #samples: 137
• #features: 102
• Train AUC: 0.75
• Hinge loss: ~0.84
• Margin stats: mean 0.16, std 0.36, min −0.66, max 0.94

The decision scores show what we want:

• Failed episodes cluster towards the left (negative margin)
• Successful episodes cluster towards the right (positive margin)
• There is clear overlap, but also a strong directional separation

This is our second confirmation:

There really is a frontier direction in this feature space where failed and successful reasoning live in different regions.

We don’t deploy this SVM; it’s a diagnostic lens to check that the geometry makes sense before we ever train the Tiny Critic.

💪 4. Training the Tiny Critic

Now we build the actual critic dataset:

1. Take the frontier feature vectors for each episode
2. Apply directionality correction so every feature follows “higher = better reasoning”
3. Standardize with StandardScaler
4. Train a logistic regression:

pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("clf", LogisticRegression(
        penalty="l2",
        C=0.5,
        solver="liblinear",
        max_iter=1000,
    )),
])

pipeline.fit(X_train, y_train)

We evaluate on a held-out set of 48 episodes (balanced 50/50):

• AUROC: ~0.68
• Brier score: ~0.26
• ECE: ~0.19

That’s not “perfect classifier” territory – and it doesn’t need to be. Remember, this is a tiny model with a few dozen features, not another 70B-parameter LLM.

What matters is that:

• it clearly beats random,
• it’s well-calibrated enough to be usable in downstream selection, and
• its weights are interpretable (“which metrics push toward ‘good’, which toward ‘suspicious’?”).

🥇 5. Downstream selection: does it actually help?

Finally, we test whether the critic’s scores actually improve a downstream task under a budget.

Setup:

• We have 48 candidate reasoning episodes for a task.

• Each is labeled correct / incorrect.

• We simulate different query budgets: keep only the top k% of episodes according to some ranking, and ask:

  “If we only trusted these, what fraction would be correct?”

We compare three selectors:

1. Random – pick k% of episodes at random
2. Top-p – pick by original model confidence
3. Critic – pick by the Tiny Critic’s score

For this run, at different budgets:

So at 20% budget:

• Random: ~49% accuracy
• Top-p: ~63%
• Tiny Critic: ~78%

That’s roughly:

• +28 points over random
• +15 points over naive confidence ranking

And remember: this gain comes from a model that:

• is ~~50KB in size,
• runs in microseconds,
• and only sees geometric features derived from existing metrics.

💯 6. Why this example matters

This single run shows the complete loop:

1. Failed vs successful reasoning form distinct clusters in frontier space.
2. A simple linear SVM sees a clear separating direction.
3. A tiny logistic critic trained on the same features generalizes with decent AUROC.
4. The critic’s ranking produces meaningful downstream gains under realistic budgets.

There’s nothing exotic hiding here: no huge extra model, no special labels beyond “correct vs incorrect”, no bespoke loss.

Just:

• metric traces → frontier geometry → tiny model → better choices.

That’s exactly the niche Tiny Critics are meant to fill.

⏺️ Apppendix 2: A Live Demo: Critic Running Beside Your Chat

Imagine you’re chatting with an AI in your browser. Every time the assistant sends a message, a tiny script:

1. extracts a small vector of reasoning metrics (stability, mid-step dips, volatility, etc.),
2. runs them through the Tiny Critic locally, and
3. paints a confidence badge on the message green, yellow, or red.

No extra API call. No extra model server. Just a dot product and a sigmoid.

Here’s a minimal demo page that shows how that looks. For now we fake the metrics, but the wiring is exactly what a real deployment would use:

<!DOCTYPE html>
<html>
<head>
  <title>Tiny Critic Demo</title>
  <style>
    body { font-family: system-ui, sans-serif; max-width: 700px; margin: 40px auto; }

    .message {
      padding: 14px 16px;
      margin: 10px 0;
      border-radius: 10px;
      position: relative;
      border: 1px solid #ddd;
    }
    .assistant { background: #f4f8ff; }

    .critic-badge {
      position: absolute;
      top: 6px;
      right: 8px;
      padding: 3px 10px;
      border-radius: 999px;
      font-size: 0.8em;
      font-weight: 600;
    }
    .high-confidence {
      background: #e6ffe6;
      color: #0a6b2f;
      border: 1px solid #a6e6a6;
    }
    .medium-confidence {
      background: #fff8e6;
      color: #8a5a00;
      border: 1px solid #f3d18a;
    }
    .low-confidence {
      background: #ffebee;
      color: #b00020;
      border: 1px solid #f5a4b0;
    }
  </style>
</head>
<body>
  <h2>Tiny Critic Browser Demo</h2>
  <p>Below, the critic is running entirely in JavaScript. Each assistant message
     gets a reasoning quality badge, but the conversation itself never leaves
     the page.</p>

  <div class="message assistant" data-quality="low">
    <p>To solve 3x + 5 = 20, subtract 5 to get 3x = 15. Then divide by 4 so x = 15/4.
       That’s close enough.</p>
  </div>

  <div class="message assistant" data-quality="high">
    <p>The equation is 3x + 5 = 20. Subtract 5: 3x = 15. Divide both sides by 3: x = 5.
       Check: 3 × 5 + 5 = 20. Correct.</p>
  </div>

  <script>
    // Tiny Critic model (would normally be loaded from a JSON file)
    const criticModel = {
      feature_names: [
        "stability", "mid_dip", "volatility", "sparsity",
        "entropy", "trend", "mid_bad_ratio", "frontier_util"
      ],
      // Example weights only   real ones come from training
      weights: [0.85, -1.2, -0.3, 0.4, -0.6, 0.9, -0.7, 1.1],
      intercept: -0.5
    };

    function sigmoid(z) {
      return 1 / (1 + Math.exp(-z));
    }

    function scoreReasoning(metrics) {
      let z = criticModel.intercept;
      for (let i = 0; i < criticModel.feature_names.length; i++) {
        const name = criticModel.feature_names[i];
        const w = criticModel.weights[i];
        const x = metrics[name] ?? 0.0;
        z += w * x;
      }
      return sigmoid(z);
    }

    // For this demo we just simulate metric vectors based on a data-quality tag
    function simulatedMetricsFor(element) {
      const quality = element.dataset.quality || "medium";
      if (quality === "high") {
        return {
          stability: 0.9, mid_dip: 0.1, volatility: 0.2,
          sparsity: 0.3, entropy: 0.4, trend: 0.8,
          mid_bad_ratio: 0.1, frontier_util: 0.9
        };
      } else if (quality === "low") {
        return {
          stability: 0.2, mid_dip: 0.8, volatility: 0.7,
          sparsity: 0.6, entropy: 0.8, trend: -0.2,
          mid_bad_ratio: 0.7, frontier_util: 0.2
        };
      }
      // medium
      return {
        stability: 0.5, mid_dip: 0.4, volatility: 0.4,
        sparsity: 0.4, entropy: 0.5, trend: 0.2,
        mid_bad_ratio: 0.4, frontier_util: 0.5
      };
    }

    function attachCriticBadge(element) {
      const metrics = simulatedMetricsFor(element);
      const score = scoreReasoning(metrics);
      const badge = document.createElement("div");
      badge.classList.add("critic-badge");

      const pct = Math.round(score * 100);
      if (score < 0.3) {
        badge.classList.add("low-confidence");
        badge.textContent = `⚠️ ${pct}%`;
      } else if (score < 0.7) {
        badge.classList.add("medium-confidence");
        badge.textContent = `⚠️ ${pct}%`;
      } else {
        badge.classList.add("high-confidence");
        badge.textContent = `✅ ${pct}%`;
      }

      element.appendChild(badge);
    }

    document.querySelectorAll(".assistant").forEach(attachCriticBadge);
  </script>
</body>
</html>

In a real system, the simulatedMetricsFor function would instead:

1. take the reasoning trace for that message,
2. run it through FrontierLens, and
3. produce the feature vector the critic expects.

Everything else the dot product, sigmoid, and badge painting would be identical.

📖 Why This Matters

This browser-native mode isn’t just a toy:

• Zero-latency verification evaluation happens at the speed of JavaScript, not the speed of an API call.
• Privacy-preserving reasoning is scored locally; no additional data leaves the browser.
• Always-on transparency every answer can carry a quiet “reasoning confidence” signal, without changing how the main model behaves.
• Ubiquitous deployment because the critic is just a few KB of JSON + JS, it can live anywhere: docs, dashboards, IDEs, notebooks, even offline tools.

And critically, the “< 50 KB” claim is structural, not marketing:

• the critic’s parameter count is tiny and fixed by its feature list,
• front-end deployment doesn’t require any heavy libraries, and
• even with generous metadata you’re still in the tens-of-KB regime.

👣 The Path Forward

Stephanie doesn’t ship this browser-native critic yet, but the current design is already compatible with it:

• The critic trainer can export feature names, scaler stats, and weights as a compact JSON artifact.
• A tiny JS helper (like the one above) can consume that artifact and attach “reasoning quality” badges to any UI that exposes reasoning traces or metrics.

As FrontierLens and FrontierIntelligence continue to stabilize and as we converge on a small, robust frontier feature set the idea of an always-on, in-browser reasoning meter stops being speculative and starts looking like the natural end-point of this work:

a tiny, ubiquitous critic that quietly watches the shape of our reasoning, wherever it happens.

📚 Glossary

Atomic Reasoning Units The smallest evaluable components of reasoning, broken down from complete thought processes into discrete steps that can be individually assessed for quality.

Band Separation A metric measuring the distance between “clearly bad” and “clearly good” reasoning in the frontier landscape. Higher values indicate better separation between reasoning quality levels.

Cohort Analysis The process of comparing two sets of reasoning traces (typically “baseline” vs “targeted”) to identify where improvements occurred and what metrics best capture those improvements.

CriticCohortAgent The component that converts Ifeatures into supervised datasets, aligning with HRM/SICQL/EBT signals, performing filtering and normalization, and creating auto-balanced samples for training.

CriticInferenceAgent The component that evaluates new reasoning traces in real time, scoring them for quality and flagging potential reasoning errors as they occur.

CriticTrainerAgent The component responsible for training the Tiny Critic model on the feature dataset extracted from reasoning traces, producing a calibrated model with metadata.

Feature Consistency A measure of how stable important features remain across different reasoning traces. High consistency indicates reliable signal for reasoning quality.

Frontier The conceptual boundary between “clearly bad” and “clearly good” reasoning in the metric space. Unlike a simple threshold, it represents a dynamic band where the most interesting reasoning occurs.

Frontier band The specific zone within the frontier landscape where reasoning transitions from bad to good. Defined by frontier_low and frontier_high thresholds, it represents the “interesting zone” where subtle reasoning errors occur.

FrontierIntelligence The meta-cognitive system that tracks critic performance across runs, analyzes metric stability, selects the most relevant frontier metric, and makes decisions about model promotion.

FrontierLens The technology that transforms reasoning traces into structured metric matrices, then analyzes them to identify the frontier band, measure quality across regions, and extract features for the Tiny Critic.

Graph of Thoughts (GoT) A paradigm where reasoning is represented as a dynamic, reusable knowledge graph rather than linear chains, allowing for more sophisticated reasoning patterns and memory.

Metric Filtering The process of selecting the most relevant metrics for reasoning evaluation, typically reducing from 20-200 metrics down to 8-30 core features that best separate good from bad reasoning.

Metric Matrix A representation of reasoning where rows = steps/scorables/tiles and columns = cognitive metrics, with each cell containing how that step scored on that metric.

Reasoning Stability A key metric measuring how consistently high-quality reasoning remains throughout a trace, without significant dips or fluctuations.

Reasoning Trajectory The path a reasoning process follows through the metric space, showing how quality evolves from beginning to end.

Scorables Stephanie’s atomic knowledge objects - the smallest evaluable units of reasoning that contain both text and feature data for analysis.

Self-Improving Feedback Loop The mechanism where critic performance informs metric selection and model promotion, creating a system that continuously improves its own reasoning evaluation capabilities.

Tiny Critic A miniature reasoning verifier (<50KB) trained on synthetic reasoning mistakes that can instantly detect broken reasoning in much larger models through geometric pattern recognition.

Tree of Thoughts (ToT) A reasoning paradigm where multiple reasoning paths are explored in parallel, allowing for more deliberate problem solving.

TopLeft Comparison A visualization technique comparing VPMs (Visual Policy Maps) between baseline and targeted reasoning to identify regions of improvement.

Visual Policy Maps (VPMs) Image-like representations of reasoning quality, where a 32×32 grid encodes clarity, coherence, directional change, and frontier alignment across a reasoning trace.

VPM Band Separation The measurable distance between baseline and targeted reasoning in Visual Policy Map space, serving as a visual validation of reasoning quality improvements.

Windowed Consistency A metric measuring how stable reasoning quality remains within specific segments of a reasoning trace, rather than just overall.

This glossary provides clear definitions of the key concepts in the Tiny Critic system, helping readers understand both the technical implementation and the conceptual framework behind this innovative approach to reasoning verification.

📚 References & Further Reading

Below are key references that informed our work on the Tiny Critic, along with additional resources for readers who want to dive deeper into the concepts behind reasoning verification.

VisPlay. Self-Evolving Vision-Language Models from Images.

This paper gave me the idea for the Frontier Lens approach

1. Wei, J., et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS.

   The seminal paper that established chain-of-thought reasoning as a critical technique for improving LLM problem-solving. Our work builds on this foundation by developing methods to verify the quality of these reasoning chains.

2. Cobbe, K., et al. (2021). Training Verifiers to Solve Math Word Problems.

   Introduces the GSM8K dataset that we use for training and evaluation. Their verifier approach inspired our work, though we take a radically different path by using geometric reasoning patterns rather than another LLM.

3. Yao, S., et al. (2023). Tree of Thoughts: Deliberate Problem Solving with Large Language Models.

   Presents the Tree of Thoughts framework that explores multiple reasoning paths. Our FrontierLens analysis complements this approach by providing objective metrics to evaluate which branches are most promising.

4. Uesato, J., et al. (2022). Solving math word problems with process- and outcome-based feedback.

   Demonstrates the value of process-based feedback for improving reasoning. Our FrontierLens implements a more automated version of this concept through metric-based analysis.

5. Lightman, H., et al. (2023). Let’s Verify Step by Step.

   A comprehensive approach to step-by-step verification using LLMs. Our Tiny Critic offers a complementary approach that’s orders of magnitude more efficient.

6. Welleck, R., et al. (2022). NaturalProofs: Mathematical Theorem Proving in Natural Language.

   While focused on mathematical proofs, this work highlights the importance of structured reasoning evaluation something our FrontierLens approach makes practical for everyday reasoning.

7. Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273-297.

   The classic SVM paper that inspired our frontier band concept. While we use logistic regression in production, the SVM validation demonstrates the geometric structure of reasoning quality.

8. Bellet, A., Habrard, A., & Sebban, M. (2015). Metric Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning.

   Provides the theoretical foundation for our approach to measuring reasoning quality in metric space.