Reference Agent Agents

Search-Solve-Prove Self-Play Loop

A verified self-play loop for agents: proposer creates tasks, solver answers with evidence, verifier checks support, and metrics become reusable training signal.

View implementation ↗

Problem Agent self-improvement needs generated practice tasks, but generated answers are useless unless evidence and verification are part of the loop.

Outcome A repeatable SSP episode format with evidence, verification, normalized metrics, visual frames, and replayable traces.

Implementation evidence

The solution is backed by inspectable code

https://github.com/ernanhughes/stephanie

This solves evidence-gated self-play for agents. Each episode has a generated challenge, an evidence-seeking solver, a verifier, and fixed metrics that can train or control later runs.

Code

def clamp01(value):
    return max(0.0, min(1.0, float(value)))


class SSPMetricsCalculator:
    VERSION = "ssp.v1"
    ORDER = [
        "ssp.reward",
        "ssp.verified",
        "ssp.curriculum_difficulty",
        "ssp.question_len",
        "ssp.answer_len",
        "ssp.evidence_count",
        "ssp.solver_steps",
        "ssp.best_score",
        "ssp.improvement",
        "ssp.search_turns",
        "ssp.f1_score",
        "ssp.format_compliance",
        "ssp.noise_tolerance",
        "ssp.rag_verification",
    ]

    def calculate(self, episode, caps):
        values = {
            "ssp.reward": clamp01(episode.reward),
            "ssp.verified": 1.0 if episode.verified else 0.0,
            "ssp.curriculum_difficulty": clamp01(episode.difficulty),
            "ssp.question_len": clamp01(len(episode.question.split()) / caps["max_question_words"]),
            "ssp.answer_len": clamp01(len(episode.answer.split()) / caps["max_answer_words"]),
            "ssp.evidence_count": clamp01(len(episode.evidence) / caps["max_evidence"]),
            "ssp.solver_steps": clamp01(episode.solver_steps / caps["max_steps"]),
            "ssp.best_score": clamp01(episode.best_score),
            "ssp.improvement": clamp01((episode.best_score - episode.base_score) / max(1 - episode.base_score, 1e-6)),
            "ssp.search_turns": clamp01(episode.search_turns / caps["max_steps"]),
            "ssp.f1_score": clamp01(episode.f1_score),
            "ssp.format_compliance": 1.0 if episode.format_compliant else 0.0,
            "ssp.noise_tolerance": clamp01(episode.noise_tolerance),
            "ssp.rag_verification": 1.0 if episode.rag_verified else 0.0,
        }
        return [values[name] for name in self.ORDER]

class SearchSolveProveLoop:
    def __init__(self, proposer, solver, verifier, metrics, memory):
        self.proposer = proposer
        self.solver = solver
        self.verifier = verifier
        self.metrics = metrics
        self.memory = memory

    async def run_episode(self, seed, caps):
        question = await self.proposer.propose(seed)
        solution = await self.solver.solve(question)
        verdict = await self.verifier.verify(
            question=question,
            answer=solution.answer,
            evidence=solution.evidence,
        )
        episode = solution.to_episode(question=question, verdict=verdict)
        metric_vector = self.metrics.calculate(episode, caps)
        self.memory.episodes.store(episode, metrics=metric_vector, metric_version=self.metrics.VERSION)
        return {"episode": episode, "metrics": metric_vector}

Usage

loop = SearchSolveProveLoop(proposer, solver, verifier, SSPMetricsCalculator(), memory)
result = await loop.run_episode(seed={"topic": "RAG verification"}, caps={
    "max_question_words": 80,
    "max_answer_words": 400,
    "max_evidence": 8,
    "max_steps": 12,
})

How it works

The Prove step is the guardrail. It turns generated practice into usable data by checking whether the answer is supported by evidence. The fixed metric order makes episodes comparable across runs and suitable for visualization or training.

Source

The implementation is part of the Stephanie/Jitter work in ernanhughes/stephanie.

Full explanation

For the full Jitter substrate, VPM filmstrips, seed vitals, controller, and verifier design, read: Search-Solve-Prove: building a place for thoughts to develop.