Reference Pipeline Evaluation

Visual Epistemic Field Generation

A visual AI pipeline for comparing populations of reasoning traces: gather good and contrasting examples, render VPM timelines, subtract fields, and rank the metrics that separate them.

View implementation ↗

Problem Reasoning scores and logs are hard to inspect at scale, especially when the useful signal is a pattern across many traces.

Outcome A reproducible process that turns metric matrices into visual fields with provenance and metric-level explanations.

Implementation evidence

The solution is backed by inspectable code

https://github.com/ernanhughes/stephanie

This solves visual debugging for reasoning systems. Instead of reading thousands of scalar scores, render trace populations as comparable images and inspect the difference between good, mixed, and opposite reasoning.

Code

class PhosAgent:
    def __init__(self, memory, zero_model, scorer, logger):
        self.memory = memory
        self.zm = zero_model
        self.scorer = scorer
        self.logger = logger
        self.metric_names = ["alignment", "coherence", "depth", "novelty"]

    async def run(self, context):
        run_id = context["pipeline_run_id"]
        goal_id = context["goal"]["id"]

        good = await self.gather_good_runs(goal_id)
        medium = self.memory.embedding.search_similarity_band(goal_id, low=0.15, high=0.80)
        opposite = self.memory.embedding.search_unrelated_scorables(goal_id, limit=300)

        datasets = {"good": good, "medium": medium, "opposite": opposite}
        vpms = await self.render_bands(run_id, datasets, context)
        context["phos_outputs"] = self.analyze_vpms(run_id, vpms)
        return context

    async def gather_good_runs(self, goal_id):
        rows = self.memory.prompts.for_goal(goal_id=goal_id, limit=300)
        return [{"response": self.strip_think_blocks(row.response_text)} for row in rows]

    async def render_bands(self, run_id, datasets, context):
        results = {}
        for label, scorables in datasets.items():
            self.zm.timeline_open(run_id=f"{run_id}:{label}")
            for scorable in scorables:
                metrics = await self.scorer.score(scorable, context)
                self.zm.timeline_add(run_id=f"{run_id}:{label}", metrics=metrics)
            results[label] = self.zm.timeline_finalize(
                run_id=f"{run_id}:{label}",
                out_path=f"data/phos/{run_id}/{label}.gif",
            )
        return results

    def analyze_vpms(self, run_id, vpms):
        good_mat = vpms["good"]["matrix"]
        opposite_mat = vpms["opposite"]["matrix"]
        meta = self.zm.generate_epistemic_field(
            pos_matrices=[good_mat],
            neg_matrices=[opposite_mat],
            output_dir=f"data/phos/{run_id}/diffs",
            metric_names=self.metric_names,
        )
        ranked = self.zm.analyze_differential_field(
            meta["diff_matrix"],
            meta["metric_names_reordered"],
            output_dir=f"data/phos/{run_id}/ranked",
        )
        return {"good_vs_opposite": meta, "ranked_metrics": ranked}

    @staticmethod
    def strip_think_blocks(text):
        return text.replace("<think>", "").replace("</think>", "")

Usage

context = await PhosAgent(memory, zero_model, scorer, logger).run({
    "pipeline_run_id": "run-2026-09-09",
    "goal": {"id": 42, "goal_text": "How can an agent improve its evidence use?"},
})

print(context["phos_outputs"]["ranked_metrics"])

How it works

The important split is by population, not by single example. Good traces form the positive field. Medium or unrelated traces form contrast classes. ZeroModel renders each population into VPM timelines, then Phos computes differential fields that show which metrics survive the comparison.

Source

The article points to the implementation in stephanie/agents/phos.py.

Full explanation

For the complete stack including ATS, MetricsWorker, VPMWorker, ZeroModel, and Phos, read: A Complete Visual Reasoning Stack: From Conversations to Epistemic Fields.