Does One Agent Plan, Execute and Judge Its Own Work? Build a Planner-Executor-Critic Architecture

A single model can often do all of these things:

• understand a task,
• decide what to do,
• execute a tool call,
• inspect the result,
• critique its own work,
• decide whether it succeeded,
• and produce the final answer.

That is convenient.

It is also a dangerous concentration of responsibilities.

If the same component creates the plan, executes it, explains why the result is good, and decides whether the job is complete, then failures become difficult to localize.

A bad plan can look like bad execution.

Bad execution can be rationalized by the critic.

A weak critic can approve the executor’s own mistakes.

A stale plan can continue driving the system even after the environment has changed.

And a model that has already invested ten steps in one trajectory has a strong tendency to interpret the latest result as progress rather than admit that the trajectory is wrong.

The next useful architecture is therefore not simply:

more agents

It is:

separate responsibilities

A common pattern is:

                     goal
                      |
                      v
                  planner
                      |
                      v
                 structured plan
                      |
                      v
                  executor
                      |
                      v
                environment
                      |
                      v
                  evidence
                      |
             +--------+--------+
             |                 |
             v                 v
           critic           verifier
             |                 |
             +--------+--------+
                      |
                      v
             continue / revise /
               replan / stop

This is often described as a planner-executor-critic architecture.

But the important idea is not the names.

The important idea is that each component has a distinct contract.

The planner proposes a route. The executor performs actions. The critic diagnoses defects. The verifier decides whether reality supports completion.

Those are different jobs.

And they should remain different even if all four roles happen to use the same underlying model.

The Search Problem This Post Solves

People building agents commonly run into questions like:

• Why does my AI agent keep changing its plan?
• Why does my planner create steps the executor cannot perform?
• Why does my executor ignore the plan?
• Why does my critic always agree with the generated answer?
• Why does my multi-agent system keep passing work back and forth?
• Why does my coding agent say the task is complete when tests still fail?
• Should I use separate planner and executor agents?
• Do I need a critic agent?
• Should the planner and executor use different models?
• How do I stop planner-executor loops?
• How do I know whether a critic agent is actually helping?

These are not primarily prompting questions.

They are control-system questions.

1. Start With the Failure, Not the Architecture

Do not begin with:

I want a planner agent,
an executor agent,
a critic agent,
a reviewer agent,
a verifier agent,
and a coordinator agent.

Begin with a measured failure.

Suppose a simple agent has a verified success rate of 68%.

You inspect failed trajectories and discover:

31% bad decomposition
26% execution mistakes
18% premature success claims
14% tool-selection errors
11% other

Now the architecture has a reason to change.

A planner may target decomposition failure.

A critic may target execution defects.

A verifier may target premature success claims.

The architectural question becomes:

Which separated responsibility removes which measured failure?

That is much stronger than:

Would multi-agent be better?

2. The Simplest Baseline: One Agent Does Everything

Consider a coding agent.

The simplest implementation might look like this:

async def solve(task, model, tools):
    state = {"task": task, "history": []}

    for step in range(20):
        response = await model.decide(state)
        action = parse_action(response)

        if action.kind == "finish":
            return action.answer

        result = await tools.execute(action)

        state["history"].append({
            "action": action,
            "result": result,
        })

    raise RuntimeError("step budget exhausted")

This can work surprisingly well.

It should remain the baseline.

The problem is that several responsibilities are hidden inside model.decide():

understand
plan
choose action
interpret result
revise strategy
judge completion

When the system fails, all of those responsibilities are entangled.

3. Separate Planning From Execution

The first separation is usually:

planner
   |
   v
plan
   |
   v
executor

The planner does not execute tools.

The executor does not silently redesign the objective.

That gives us two inspectable contracts.

A Structured Plan

Avoid prose-only plans like:

First inspect the repository, then probably update the file,
then run some tests and fix anything that looks wrong.

Instead represent the plan as data.

from dataclasses import dataclass, field
from typing import Literal

StepStatus = Literal[
    "pending",
    "running",
    "passed",
    "failed",
    "blocked",
    "skipped",
]

@dataclass
class PlanStep:
    id: str
    objective: str
    action_family: str
    depends_on: list[str] = field(default_factory=list)
    expected_evidence: list[str] = field(default_factory=list)
    status: StepStatus = "pending"

@dataclass
class Plan:
    goal: str
    version: int
    steps: list[PlanStep]

Now the runtime can inspect the plan independently of the model.

It can ask:

• Are IDs unique?
• Do dependencies exist?
• Is the dependency graph acyclic?
• Does each step request an available capability?
• Does every terminal path have a verification step?
• Is the plan too large for the budget?

These are ordinary software checks.

4. The Planner Contract

The planner should not own the entire system.

Its responsibility is narrower:

input:
    goal
    capabilities
    known constraints
    current verified state

output:
    candidate plan

That is all.

A useful interface might look like:

from typing import Protocol

class Planner(Protocol):
    async def create_plan(
        self,
        *,
        goal: str,
        capabilities: list[str],
        facts: dict,
    ) -> Plan:
        ...

The planner does not get to:

• grant itself capabilities,
• fabricate successful execution,
• redefine the user’s goal,
• claim completion,
• or bypass verification.

The runtime owns those boundaries.

5. Validate Plans Before Execution

A planner can hallucinate steps just like an agent can hallucinate tools.

For example:

1. inspect repository
2. call deploy_to_production
3. run hidden integration test
4. merge PR

But perhaps the runtime only exposes:

read_file
search_code
edit_file
run_tests

Then the plan is invalid before execution begins.

A simple validator:

class PlanValidationError(Exception):
    pass


def validate_plan(plan: Plan, capabilities: set[str]) -> None:
    ids = [step.id for step in plan.steps]

    if len(ids) != len(set(ids)):
        raise PlanValidationError("duplicate step IDs")

    id_set = set(ids)

    for step in plan.steps:
        unknown = set(step.depends_on) - id_set
        if unknown:
            raise PlanValidationError(
                f"step {step.id} has unknown dependencies: {sorted(unknown)}"
            )

        if step.action_family not in capabilities:
            raise PlanValidationError(
                f"unsupported action family: {step.action_family}"
            )

The model proposes.

The runtime validates.

That rule survives every architecture in this series.

6. The Executor Contract

The executor has a different job.

It receives one actionable unit of work and attempts it using the available environment.

@dataclass
class StepResult:
    step_id: str
    success: bool
    evidence: dict
    error: str | None = None


class Executor(Protocol):
    async def execute_step(
        self,
        *,
        step: PlanStep,
        state: dict,
    ) -> StepResult:
        ...

The executor should not silently rewrite:

"Run tests"

into:

"Skip tests and declare success because the patch looks correct."

If execution cannot satisfy the planned step, it should report that explicitly.

For example:

StepResult(
    step_id="run_tests",
    success=False,
    evidence={
        "command": "pytest -q",
        "exit_code": 1,
        "failing_tests": [
            "test_repository_loader",
            "test_plan_validation",
        ],
    },
    error="2 tests failed",
)

That result becomes evidence for the next control decision.

7. Executor Drift

One of the most common planner-executor failures is executor drift.

The planner says:

inspect failing tests

The executor decides:

rewrite the implementation

The planner says:

edit one file

The executor changes twelve files.

The planner says:

perform a read-only diagnostic

The executor performs a side effect.

This is not necessarily malicious or irrational.

It often happens because the executor is given an underspecified objective and enough autonomy to reinterpret it.

Detecting Executor Drift

Every step should define an execution envelope.

@dataclass
class ExecutionEnvelope:
    allowed_tools: set[str]
    allowed_paths: set[str]
    max_tool_calls: int
    side_effects_allowed: bool

Then runtime checks can reject actions outside the envelope.

def authorize_action(action, envelope: ExecutionEnvelope):
    if action.tool not in envelope.allowed_tools:
        raise PermissionError("tool outside execution envelope")

    if action.side_effect and not envelope.side_effects_allowed:
        raise PermissionError("side effect not allowed for this step")

Again:

model autonomy
      !=
runtime authority

8. Why Add a Critic?

Suppose execution completes.

The result may still be poor.

A coding patch can compile but violate the architectural requirement.

A research summary can cite sources but misrepresent the main claim.

A browser workflow can fill every field but put the wrong data in one field.

A support agent can resolve the ticket operationally while violating policy.

This is where a critic can help.

But only if its responsibility is concrete.

9. A Critic Is Not a Generic “Reviewer”

This prompt is weak:

Review the result and tell me if it is good.

It encourages generic approval language.

A better critic has a defect contract.

@dataclass
class Critique:
    defect_type: str
    severity: str
    evidence: list[str]
    affected_requirement: str | None
    suggested_action: str | None
    confidence: float

And the critic receives explicit criteria.

Check only for:

- violation of requested behavior
- unhandled failure paths
- regression risk
- evidence missing for claimed completion

Now the critic’s output can affect control flow.

10. Critics Need Evidence Too

A critic saying:

The implementation looks robust.

is weak evidence.

A critic saying:

Potential defect:
`load_repository()` catches `Exception` and converts all failures to an
empty repository. This can hide authentication and filesystem errors.
Evidence: src/repository.py lines 71-78.

is more useful.

The critic should point to observable features.

claim
  +
location / evidence
  +
criterion violated

That makes the critique inspectable.

11. Critic Agreement Bias

A common failure is that the critic agrees with the executor because both share:

• the same model,
• the same prompt context,
• the same assumptions,
• the same retrieved evidence,
• and the same framing.

Adding another model call does not guarantee independent judgment.

This architecture:

model A -> executor
model A -> critic

can easily become:

same failure mode twice

The correct question is not:

Did we add a critic?

It is:

Did the critic detect additional verified defects?

12. Measure Marginal Critic Value

Suppose your baseline executor produces 100 outputs.

External evaluation identifies 40 defective outputs.

A critic detects 25 of them.

That is useful.

Now add a second critic.

If the second critic detects the same 25 defects and only one additional verified defect, its marginal value may be low.

Track:

critic recall
critic precision
unique verified defects
false-positive critique rate
cost per additional verified defect

A simple record:

@dataclass
class CriticMetrics:
    true_defects: int
    detected_defects: int
    false_positives: int
    unique_defects_vs_previous_critics: int
    model_calls: int
    cost: float

This prevents critic armies.

13. Critic Is Still Not Verifier

This distinction is critical.

critic:
    "I think this implementation has no obvious defects."

is not:

verifier:
    "All required tests passed against commit abc123."

A critic reasons about quality.

A verifier checks externally observable criteria.

The architecture should therefore be:

planner
   ↓
executor
   ↓
critic
   ↓
possible revision
   ↓
verifier
   ↓
PASS / FAIL / UNKNOWN

Not:

critic says good
   ↓
success

14. The Verifier Contract

The verifier needs explicit success criteria.

from dataclasses import dataclass
from typing import Literal

VerificationStatus = Literal["PASS", "FAIL", "UNKNOWN"]

@dataclass
class VerificationResult:
    status: VerificationStatus
    evidence: dict
    state_id: str
    missing_checks: list[str]

For coding agents:

state_id = git tree hash

For data systems:

state_id = dataset version

For deployments:

state_id = deployment ID

For browser workflows:

state_id = transaction / form submission ID

Verification must bind to the exact state being accepted.

15. A Full Planner-Executor-Critic Runtime

Here is a simplified architecture.

from dataclasses import dataclass, field

@dataclass
class RunState:
    goal: str
    plan: Plan | None = None
    step_results: list[StepResult] = field(default_factory=list)
    critiques: list[Critique] = field(default_factory=list)
    replans: int = 0
    revisions: int = 0


class PlannerExecutorCriticRuntime:
    def __init__(
        self,
        *,
        planner,
        executor,
        critic,
        verifier,
        max_replans=2,
        max_revisions=3,
    ):
        self.planner = planner
        self.executor = executor
        self.critic = critic
        self.verifier = verifier
        self.max_replans = max_replans
        self.max_revisions = max_revisions

    async def run(self, goal: str, capabilities: set[str]):
        state = RunState(goal=goal)

        state.plan = await self.planner.create_plan(
            goal=goal,
            capabilities=sorted(capabilities),
            facts={},
        )

        validate_plan(state.plan, capabilities)

        while True:
            step = next_ready_step(state.plan)

            if step is None:
                break

            result = await self.executor.execute_step(
                step=step,
                state=state.__dict__,
            )

            state.step_results.append(result)

            if not result.success:
                decision = decide_recovery(state, result)

                if decision == "replan":
                    if state.replans >= self.max_replans:
                        break

                    state.replans += 1
                    state.plan = await self.planner.create_plan(
                        goal=goal,
                        capabilities=sorted(capabilities),
                        facts=collect_verified_facts(state),
                    )
                    validate_plan(state.plan, capabilities)
                    continue

                break

            mark_step_passed(state.plan, step.id)

        critique = await self.critic.review(
            goal=goal,
            plan=state.plan,
            results=state.step_results,
        )

        state.critiques.append(critique)

        verification = await self.verifier.verify(
            goal=goal,
            state=state,
        )

        return state, verification

This example is intentionally incomplete.

Production systems need:

• authorization,
• timeouts,
• tool budgets,
• cost budgets,
• retries,
• concurrency control,
• state binding,
• provenance,
• recovery policies,
• persistence,
• failure telemetry,
• and explicit termination reasons.

But the important architecture is visible.

16. The Coordinator Should Mostly Be Code

Many multi-agent systems add a coordinator model.

planner
   ↓
coordinator
   ↓
executor
   ↓
coordinator
   ↓
critic
   ↓
coordinator
   ↓
verifier

Sometimes that is justified.

Often it is not.

If the routing rule is known:

if plan_missing:
    call_planner()
elif ready_step:
    call_executor()
elif execution_complete:
    call_critic()
elif critique_requires_revision:
    call_reviser()
else:
    call_verifier()

then ordinary code is more reliable than asking another model:

Which agent should speak next?

Use model routing only when the choice itself is genuinely semantic or uncertain.

17. Multi-Agent Does Not Require Multiple Models

A planner-executor-critic system can use one underlying model.

same model
  ├→ planner contract
  ├→ executor contract
  └→ critic contract

The benefit may come from separated state, permissions and objectives rather than model diversity.

Alternatively:

small local model -> planner
frontier model    -> executor
learned scorer    -> critic
pytest            -> verifier

Or:

deterministic DAG planner
LLM executor
static analyzer critic
integration test verifier

The architecture is about responsibilities.

Not personalities.

18. Planner Drift

Planner drift happens when the planner gradually changes the intended objective.

The user asks:

Fix the failing unit test without changing public behavior.

After two replans the system is effectively solving:

Refactor the module to make the tests easier to satisfy.

That is a different goal.

The runtime should preserve an immutable goal contract.

@dataclass(frozen=True)
class GoalContract:
    objective: str
    constraints: tuple[str, ...]
    success_criteria: tuple[str, ...]

Replanning can change the route.

It cannot silently change the contract.

19. Stale Plans

Plans are hypotheses about the future.

The environment can invalidate them.

Suppose the plan says:

1. edit module A
2. run tests
3. update module B

But after step 1, tests reveal module B is unrelated.

Continuing blindly wastes work.

A plan therefore needs replan triggers.

Examples:

unexpected environment state
new constraint discovered
required tool unavailable
critical assumption falsified
step output differs from expected evidence
verification failure changes diagnosis

Do not replan merely because the model feels uncertain.

Tie replanning to observable events where possible.

20. Replanning Without Losing Completed Work

A bad implementation throws away the entire plan.

old plan -> failure -> new plan from scratch

That can cause repeated work.

Instead preserve verified state.

completed facts
completed artifacts
successful tool outputs
validated constraints

Then replan only the unresolved portion.

facts = collect_verified_facts(state)

new_plan = await planner.create_plan(
    goal=goal,
    capabilities=capabilities,
    facts=facts,
)

This makes replanning incremental rather than amnesic.

21. Circular Self-Approval

One subtle failure looks like this:

planner proposes plan
executor follows plan
critic judges execution against plan

Everything can appear internally consistent while the original user goal is still unmet.

Why?

Because the critic is evaluating against the planner’s interpretation rather than the external success criteria.

This is circular self-approval.

The fix is simple conceptually:

user goal
  ↓
independent success criteria

All roles reference that contract.

The planner does not own it.

22. Handoff Loss

Every agent handoff can lose information.

planner -> executor
executor -> critic
critic -> reviser
reviser -> verifier

If each component receives a prose summary of the previous component, information degrades.

Prefer shared structured state.

@dataclass
class SharedTaskState:
    goal: GoalContract
    plan: Plan
    verified_facts: dict
    execution_results: list[StepResult]
    critiques: list[Critique]
    artifacts: dict

Then each role reads the same canonical state.

Do not make each model reconstruct reality from another model’s summary.

23. Excessive Handoffs

A common advanced-agent failure is architecture inflation.

user
 ↓
manager
 ↓
planner
 ↓
researcher
 ↓
executor
 ↓
critic
 ↓
reviewer
 ↓
judge
 ↓
reviser
 ↓
verifier
 ↓
manager

That looks sophisticated.

It may simply multiply:

• latency,
• token usage,
• context translation,
• routing errors,
• duplicated reasoning,
• and failure opportunities.

Track handoff value.

For each role ask:

What unique failure does this role catch?

If the answer is unclear, remove the role and benchmark again.

24. The Failure Matrix

A useful way to debug this architecture is to classify failures by responsibility.

This is one of the main reasons to separate roles.

Failure attribution becomes possible.

25. Application: Coding Agents

Planner-executor-critic architectures fit coding agents well because software already provides strong external evidence.

A possible architecture:

issue / request
   ↓
planner
   ↓
repository inspection plan
   ↓
executor
   ├→ read files
   ├→ edit files
   ├→ run tests
   └→ inspect diff
   ↓
critic
   ├→ architecture violations
   ├→ regression risk
   └→ missing edge cases
   ↓
verifier
   ├→ tests
   ├→ type checks
   ├→ lint
   ├→ build
   └→ acceptance criteria

Strong planner inputs include:

• repository graph,
• changed files,
• failing tests,
• dependency graph,
• issue constraints,
• available tools.

Strong critic evidence includes:

• diff,
• AST structure,
• static analysis,
• test failures,
• changed dependency boundaries.

Strong verification includes:

• exact commit/tree hash,
• test suite results,
• build artifacts,
• behavior-level acceptance tests.

26. Application: Code Review

A reviewer system may use:

change classifier
   ↓
risk planner
   ↓
specialist critics
   ├→ correctness
   ├→ security
   ├→ concurrency
   └→ API compatibility
   ↓
external checks
   ↓
review synthesis

Notice that the planner may not be planning implementation work.

It may be planning review coverage.

That is an important application of the same architecture.

27. Application: Research Agents

Research agents often fail because one component both forms the hypothesis and evaluates the sources supporting it.

A better split:

research question
   ↓
planner
   ├→ subquestions
   ├→ evidence needs
   └→ source requirements
   ↓
executor
   ├→ search
   ├→ retrieve
   └→ extract evidence
   ↓
critic
   ├→ unsupported claims
   ├→ source mismatch
   └→ contradictory evidence
   ↓
verifier
   ├→ source provenance
   ├→ quote / claim alignment
   └→ primary-source checks

The critic can reason about evidence quality.

The verifier confirms whether the cited source actually supports the claim.

28. Application: Incident Response

Incident response is another strong fit.

alert
 ↓
planner
   ├→ identify hypotheses
   ├→ order diagnostics
   └→ define rollback criteria
 ↓
executor
   ├→ inspect logs
   ├→ inspect metrics
   ├→ query deployments
   └→ run read-only diagnostics
 ↓
critic
   ├→ alternative root causes
   ├→ unsafe actions
   └→ evidence gaps
 ↓
verifier
   ├→ service health
   ├→ error rate
   └→ rollback / deployment state

The side-effect boundary matters enormously here.

Diagnostics can be broad.

Production mutation should remain tightly authorized.

29. Application: Customer Support

A support workflow might use:

customer request
   ↓
planner
   ├→ identify required information
   ├→ policy checks
   └→ resolution path
   ↓
executor
   ├→ retrieve account
   ├→ inspect order
   ├→ calculate eligibility
   └→ perform allowed action
   ↓
critic
   ├→ policy violation
   ├→ missing information
   └→ inappropriate resolution
   ↓
verifier
   ├→ refund state
   ├→ ticket state
   └→ confirmation ID

The verifier checks actual system state.

Not the assistant’s sentence:

Your refund has been processed.

30. Application: Data and Analytics Agents

A data agent may split responsibilities like this:

analysis request
   ↓
planner
   ├→ required datasets
   ├→ transformations
   └→ checks
   ↓
executor
   ├→ SQL
   ├→ transformations
   └→ statistics
   ↓
critic
   ├→ leakage
   ├→ aggregation errors
   ├→ denominator mistakes
   └→ unsupported interpretation
   ↓
verifier
   ├→ row counts
   ├→ schema
   ├→ constraints
   └→ reproducible query/output hash

Again, role separation maps cleanly onto normal engineering practice.

31. Application: Browser Automation

Browser agents often benefit from a planner-executor split because page-level actions are cheap but wrong commitments can become expensive.

objective
 ↓
planner
   ├→ page sequence
   ├→ required fields
   └→ stop before irreversible submission
 ↓
executor
   ├→ navigate
   ├→ click
   ├→ type
   └→ extract
 ↓
critic
   ├→ wrong field
   ├→ unexpected page state
   └→ missing confirmation
 ↓
verifier
   ├→ confirmation page
   ├→ transaction ID
   └→ external account state

A strong safety rule is:

search and prepare freely
commit deliberately
verify externally

32. When Should Planner and Executor Use Different Models?

Sometimes the planner benefits from a strong model while execution can use a cheaper model.

frontier planner
   ↓
local executor

Sometimes the opposite is sensible.

Planning may be mostly deterministic, while difficult implementation needs the strongest model.

deterministic planner
   ↓
frontier executor

Sometimes both can be local until verification fails.

local planner
   ↓
local executor
   ↓
verify
   ↓
FAIL / UNKNOWN
   ↓
frontier escalation

Benchmark these variants.

Do not assume role prestige maps to model size.

33. Cost-Aware Role Allocation

Suppose you have:

local model:     cheap, fast
frontier model:  expensive, stronger
static analyzer: deterministic
unit tests:      deterministic

A sensible architecture may be:

local planner
   ↓
local executor
   ↓
static critic
   ↓
tests
   ↓
if uncertainty / failure remains
   ↓
frontier critic or executor

This is not just multi-agent orchestration.

It is adaptive compute allocation.

34. Planner Confidence Is Not Enough

A planner may provide:

{
  "confidence": 0.94
}

That number is not a reason to trust the plan.

Plan quality should be measured using outcomes.

Track:

plan validity rate
steps executed successfully
replan rate
unnecessary replan rate
planner-caused failure rate
verified success conditioned on plan type

Calibration matters more than confidence prose.

35. Critic Confidence Is Not Enough Either

Likewise:

{
  "severity": "high",
  "confidence": 0.97
}

may still be wrong.

Measure critic predictions against later verified outcomes.

critic precision
critic recall
severity calibration
false-positive rate
marginal defect discovery

The critic is a model component.

It should be evaluated like one.

36. Planner-Executor-Critic vs Mixture of Experts

These architectures solve different problems.

Mixture of Experts asks:

Who should do this work?

Planner-executor-critic asks:

Which responsibility is active now?

You can combine them.

planner
   ↓
router
   ├→ code executor
   ├→ research executor
   └→ browser executor
   ↓
critic router
   ├→ security critic
   ├→ correctness critic
   └→ policy critic
   ↓
verifier

But do not combine them merely because you can.

Each dimension multiplies complexity.

37. Planner-Executor-Critic vs MCTS

MCTS manages search allocation over branches.

Planner-executor-critic manages responsibilities during one trajectory.

A planner can generate candidate strategies that MCTS explores.

An executor can simulate branches.

A critic can provide heuristic value estimates.

A verifier can score terminal outcomes.

But MCTS is not required for planner-executor separation.

Most systems should try the simpler architecture first.

38. Planner-Executor-Critic vs Workflow

If the sequence is always:

parse input
 ↓
query database
 ↓
apply rule
 ↓
generate response

then use a workflow.

Do not use a planner.

Planner-executor architectures are useful when:

which steps are needed

or:

which order should they run

changes with the task or environment.

39. A Useful Escalation Ladder

Start with:

single model call

If execution is fixed:

fixed workflow

If next action depends on observations:

agent loop

If decomposition fails:

add planner

If execution quality fails:

add targeted critic

If completion claims are unreliable:

add external verifier

If different task types need different capabilities:

add expert routing

If multiple trajectories must be explored:

add search

That is a much healthier progression than beginning with six agents.

40. Benchmark the Architecture

Compare at least:

A. one-shot model
B. simple agent loop
C. planner + executor
D. planner + executor + critic
E. planner + executor + critic + verifier

For each system measure:

verified success rate
planner failure rate
executor failure rate
critic precision / recall
false-success rate
replan rate
steps
model calls
tool calls
latency
cost
cost per verified success

Then calculate marginal gain.

marginal_gain = (
    verified_success_new - verified_success_baseline
)

And marginal cost.

marginal_cost = cost_new - cost_baseline

Do not merely report that the advanced system used more reasoning.

Report whether it solved more tasks.

41. Role Ablations

Ablations are particularly useful here.

Remove the planner.

What changes?

Remove the critic.

What changes?

Replace the LLM critic with deterministic checks.

What changes?

Use the same model for planner/executor versus different models.

What changes?

Remove replanning.

What changes?

This tells you which mechanism is actually earning its keep.

42. Failure Injection

Do not wait for natural failures.

Inject them.

For coding agents:

introduce failing test
remove required tool
return stale file content
create conflicting dependency
make build pass while acceptance test fails

For browser agents:

change element ID
insert unexpected confirmation step
return partial form state
simulate expired session

For data agents:

schema drift
missing column
stale dataset
incorrect row count

Then measure whether the architecture detects and recovers.

43. Trajectory Logging

Advanced systems need trajectory-level observability.

A useful event record:

@dataclass
class AgentEvent:
    run_id: str
    role: str
    event_type: str
    input_state_id: str
    output_state_id: str | None
    model: str | None
    tool: str | None
    latency_ms: int
    cost: float
    metadata: dict

Events might include:

plan_created
plan_validated
step_started
step_completed
critic_finding
revision_requested
replan_triggered
verification_passed
verification_failed

Without this, multi-role systems become almost impossible to debug.

44. Handoff Count Is a Metric

Track:

handoffs per successful task
handoffs per failed task

If failures involve far more handoffs, the architecture may be oscillating.

Also track role transitions.

planner -> executor
executor -> planner
planner -> executor
executor -> planner

Repeated transitions can expose planner instability.

executor -> critic
critic -> executor
executor -> critic
critic -> executor

Repeated transitions may expose unresolved critique loops.

45. Termination Reasons Must Be Explicit

Do not return only:

finished = True

Use named outcomes.

VERIFIED_SUCCESS
VERIFIED_FAILURE
UNKNOWN
PLAN_INVALID
EXECUTION_BLOCKED
REPLAN_BUDGET_EXHAUSTED
REVISION_BUDGET_EXHAUSTED
TOOL_BUDGET_EXHAUSTED
TIME_BUDGET_EXHAUSTED
AUTHORIZATION_DENIED

This matters enormously for production analysis.

46. Do Not Let the Planner Grade the Executor

This is tempting:

planner creates expected output
executor acts
planner decides whether execution matched expectation

But the planner may simply defend its own decomposition.

Keep evaluation independent when possible.

Even if independence is only architectural rather than model-level, preserve the contracts.

47. Do Not Let the Critic Rewrite the Goal

A critic should identify defects against requirements.

It should not transform:

user requested minimal bug fix

into:

rewrite the subsystem because it would be cleaner

Preservation constraints matter.

@dataclass(frozen=True)
class CriticContract:
    allowed_defect_types: tuple[str, ...]
    preservation_constraints: tuple[str, ...]

Critique is not authorization.

48. Do Not Let the Executor Treat Critique as Truth

A critic can hallucinate.

So the executor should not blindly implement every critique.

A better flow:

critic finding
   ↓
validate evidence
   ↓
material defect?
   ├→ no -> ignore
   └→ yes -> revise

For code, a critic might claim:

this function is unused

Before deleting it, search references.

Use tools to test critic claims.

49. Critic-Driven Evidence Acquisition

The critic does not need to produce a final judgment immediately.

It can say:

I cannot determine whether this is safe without knowing whether
`parse_config()` is used by external callers.

That can trigger:

search references
inspect public API
run compatibility test

This is more useful than forcing a binary critique from incomplete evidence.

50. Separate UNKNOWN From FAIL

Suppose the verifier cannot access the integration environment.

That is not:

PASS

And it is not necessarily:

FAIL

It may be:

UNKNOWN

The runtime can then decide whether to:

• escalate,
• request human review,
• retry later,
• gather more evidence,
• or stop safely.

This distinction prevents false certainty.

51. Real Software Pattern: Compiler Architecture

There is a useful analogy here.

A compiler does not ask one giant component to:

parse
optimize
generate code
validate syntax
execute program
verify behavior

It has stages with contracts.

Agent runtimes benefit from the same principle.

intent
 ↓
plan representation
 ↓
validated actions
 ↓
execution
 ↓
evidence
 ↓
verification

The goal is not to imitate a human team.

The goal is to create inspectable software boundaries.

52. The Architectural Smell Test

If your system has agents named:

BossAgent
ManagerAgent
SupervisorAgent
ReviewerAgent
JudgeAgent
MasterAgent

ask what each one uniquely controls.

Names are not architecture.

Contracts are architecture.

A good role definition should answer:

What input does this role receive?
What output is it allowed to produce?
What authority does it have?
What evidence can it observe?
What failure does it target?
How is its value measured?

If those answers are fuzzy, the role probably is too.

53. A Compact Production Architecture

A robust default may look like this:

                    immutable goal contract
                            |
                            v
                         planner
                            |
                     validated plan
                            |
                            v
                         executor
                            |
                     structured evidence
                            |
             +--------------+--------------+
             |                             |
             v                             v
      deterministic checks             critic
             |                             |
             +--------------+--------------+
                            |
                   revision if justified
                            |
                            v
                         verifier
                            |
              +-------------+-------------+
              |             |             |
              v             v             v
            PASS          FAIL         UNKNOWN

Notice what is missing:

coordinator LLM

Unless routing genuinely requires semantic judgment, ordinary runtime code can coordinate the roles.

54. Production Checklist

Before deploying a planner-executor-critic architecture, verify that:

[ ] goal contract is immutable
[ ] plans are structured and validated
[ ] executor permissions are explicit
[ ] executor drift is observable
[ ] replanning has concrete triggers
[ ] completed verified work survives replanning
[ ] critic has a narrow defect contract
[ ] critic findings require evidence
[ ] critic precision/recall are measured
[ ] critic is not treated as verifier
[ ] final verification uses external evidence
[ ] verification is bound to exact state
[ ] UNKNOWN is preserved
[ ] handoffs are logged
[ ] replan/revision budgets exist
[ ] termination reasons are explicit
[ ] each role has measurable marginal value

If several of these are missing, adding more agents will usually make the system harder to understand rather than more capable.

55. The Evidence Rule

The architecture earns its complexity only if it improves verified outcomes.

Suppose:

simple agent
verified success: 78%
median latency:    8s
cost/task:         $0.04

And:

planner-executor-critic
verified success: 79%
median latency:    31s
cost/task:         $0.22

The advanced system may not be justified.

But if:

planner-executor-critic
verified success: 92%
median latency:    15s
cost/task:         $0.09

and the increase comes specifically from fewer decomposition and regression failures, then the architecture has earned its place.

That is the standard.

Not how sophisticated the diagram looks.

56. What We Have Built So Far

The advanced series now has several independent mechanisms.

Chain of Thought
    intermediate computation

Self-Consistency
    repeated sampling + disagreement

Tree of Thoughts
    branching intermediate reasoning

MCTS
    adaptive search-compute allocation

Mixture of Experts
    capability specialization + routing

Planner-Executor-Critic
    responsibility separation + control

These are not steps on one capability ladder.

They are dimensions.

A production system may need one of them.

Or several.

Or none.

57. The Next Failure

Separating planner, executor and critic creates a new question.

What happens when multiple agents or specialists disagree?

Suppose:

security critic: reject
performance critic: accept
correctness critic: accept
planner: proceed
verifier: UNKNOWN

Or:

agent A says hypothesis X
agent B says hypothesis Y
agent C attacks both

Simply taking a majority vote can reproduce the self-consistency problem we already saw.

The next architecture therefore needs to ask:

When does adversarial review or multi-agent debate produce genuinely new evidence rather than more correlated model opinions?

That is the subject of the next post.

Conclusion

Planner-executor-critic systems are useful because they separate responsibilities that are easy to conflate inside one model call.

The planner decides what should happen.

The executor attempts the work.

The critic looks for specific defects.

The verifier checks whether reality supports success.

The runtime controls the boundaries between them.

The most important lesson is not:

use more agents

It is:

Give every responsibility a contract, every handoff a structured state, every claim an evidence path, and every additional role a measurable reason to exist.

That is what turns a collection of prompts into an engineering system.

In the next post we will move from role separation to disagreement itself:

Advanced Agents From First Principles 07: Do Your Agents Agree Too Easily? Use Adversarial Review and Multi-Agent Debate Without Confusing Debate With Truth.