Reference Pipeline Evaluation

Model Gap Calibration and Routing

A model-agnostic evaluation pipeline for turning score gaps between two models into calibration curves, routing thresholds, visual diagnostics, and provenance.

View implementation ↗

Problem A cheaper model may be good enough for many cases, but raw scores do not show where it disagrees with a stronger model or when to escalate.

Outcome A repeatable pipeline that aligns model outputs into shared metrics, computes delta fields, calibrates scores, and emits routing policy.

Implementation evidence

The solution is backed by inspectable code

https://github.com/ernanhughes/stephanie

This solves practical model routing: use the structure of disagreement between two models as an operating signal instead of asking which model is globally better.

Code

from dataclasses import dataclass


@dataclass
class ScoreRow:
    item_id: str
    dimension: str
    model: str
    score01: float
    uncertainty01: float = 0.0
    ood01: float = 0.0


def align_scores(rows):
    aligned = {}
    for row in rows:
        aligned.setdefault((row.item_id, row.dimension), {})[row.model] = row
    return aligned


def compute_delta_records(rows, strong_model="hrm", small_model="tiny"):
    records = []
    for (item_id, dimension), pair in align_scores(rows).items():
        if strong_model not in pair or small_model not in pair:
            continue
        strong = pair[strong_model]
        small = pair[small_model]
        records.append({
            "item_id": item_id,
            "dimension": dimension,
            "strong_score01": strong.score01,
            "small_score01": small.score01,
            "delta01": strong.score01 - small.score01,
            "uncertainty01": small.uncertainty01,
            "ood01": small.ood01,
        })
    return records

class RoutingPolicy:
    def __init__(self, delta_threshold=0.25, uncertainty_threshold=0.6, ood_threshold=0.7):
        self.delta_threshold = delta_threshold
        self.uncertainty_threshold = uncertainty_threshold
        self.ood_threshold = ood_threshold

    def route(self, record):
        if abs(record["delta01"]) >= self.delta_threshold:
            return "escalate_to_strong_model"
        if record["uncertainty01"] >= self.uncertainty_threshold:
            return "escalate_to_strong_model"
        if record["ood01"] >= self.ood_threshold:
            return "escalate_to_strong_model"
        return "use_small_model"


def summarize_routing(delta_records, policy):
    decisions = [policy.route(record) for record in delta_records]
    small_count = decisions.count("use_small_model")
    return {
        "total": len(decisions),
        "small_model_usage_rate": small_count / max(len(decisions), 1),
        "escalation_rate": 1 - (small_count / max(len(decisions), 1)),
        "thresholds": {
            "delta": policy.delta_threshold,
            "uncertainty": policy.uncertainty_threshold,
            "ood": policy.ood_threshold,
        },
    }

Usage

rows = run_dual_pass_scoring(items, models=["hrm", "tiny"])
delta_records = compute_delta_records(rows)
policy = RoutingPolicy(delta_threshold=0.25, uncertainty_threshold=0.6, ood_threshold=0.7)
summary = summarize_routing(delta_records, policy)

How it works

The key is to force both models into Shared Canonical Metrics before comparison. Once scores, uncertainty, and OOD signals are aligned, the gap becomes measurable. Calibration then maps the small model toward the stronger model, while routing thresholds decide when the small model is safe to use.

Source

The implementation is part of the Stephanie GAP component in ernanhughes/stephanie.

Full explanation

For the full HRM/Tiny comparison, SCM plugin architecture, delta-field visualization, topology analysis, calibration, and provenance model, read: The Space Between Models Has Holes: Mapping the AI Gap.