# Advanced Agents From First Principles 07: Do Your Agents Agree Too Easily? Use Adversarial Review and Multi-Agent Debate Without Confusing Debate With Truth
# Book: Advanced Agents From First Principles
# Chapter URL: https://programmer.ie/books/advanced-agents-from-first-principles/07-chapter/
# Prose fingerprint (sha256, normalized): f87603872229137147869c1fec3a55e5567114a18d8c1dbebc5448fc76c9f1a3
# Snapshot generated at build time by Hugo. This is the complete rendered prose
# of the chapter; it is not a transcript of the source file.

A multi-agent system can look sophisticated while every agent quietly repeats the same mistake.
That is one of the most dangerous failure modes in advanced agent architectures.
You ask one model to solve the problem.
Then you ask a second model to review it.
Then a third model judges the disagreement.
Three calls later, the system sounds more confident than before.
But if all three agents share the same blind spot, the extra machinery has not created independent evidence.
It has created correlated confidence.
This post is about a more disciplined use of adversarial review and multi-agent debate.
The central rule is:
Debate is useful for generating challenges. It is not a source of truth.
A good adversarial system does not ask agents to argue until one sounds convincing.
It asks them to expose assumptions, produce counterexamples, identify missing evidence, and route disputed claims toward stronger verification.
The architecture is closer to this:
proposal ↓ adversarial review ↓ claims / objections / counterexamples ↓ evidence requests ↓ external checks ↓ adjudication ↓ PASS / FAIL / UNKNOWN The final authority is still the environment, tests, source data, primary evidence, deterministic constraints, or a verifier with a clearly defined contract.
Not the loudest agent.
The searchable problem: “Why do my agents agree even when they are wrong?” Developers often add more agents after observing unreliable outputs.
The intuition is understandable:
one model can be wrong therefore multiple models should correct one another Sometimes they do.
But only when the additional agents contribute something genuinely different:
different evidence, different capabilities, different failure sensitivities, different models, different tools, different search paths, or deliberately different assumptions. If five agents receive the same context, use the same model family, optimize for the same style of answer, and see the same retrieved evidence, their errors can be strongly correlated.
Then majority agreement tells you very little.
This is the same warning we encountered with self-consistency:
Agreement measures the distribution of generated answers. It does not prove that the shared answer is correct.
Adversarial review becomes valuable when it changes the information available to the runtime.
What adversarial review is actually for The simplest useful role of an adversarial agent is not to produce a competing final answer.
It is to attack specific claims in the current proposal.
Suppose a coding agent proposes:
The bug is caused by the connection pool timing out. A weak review asks:
Do you agree? A stronger adversarial review asks:
What evidence would falsify the timeout hypothesis? Which logs should contain that evidence? What alternative failure modes fit the same symptoms? What assumption is the proposed diagnosis making about retry behavior? Now the critic is producing useful work even before we know whether its own alternative explanation is correct.
It is generating tests of the proposal.
That distinction matters.
Role diversity is not prompt diversity A common multi-agent architecture looks like this:
optimist agent pessimist agent security agent performance agent judge agent That can work.
But the labels themselves do not create independent capability.
If every role is the same model with the same context and the same underlying information, the system may still collapse toward one shared interpretation.
Useful diversity comes from changing the agent’s actual information or responsibility.
For example:
implementation reviewer -> repository diff + tests security reviewer -> taint analysis + auth boundaries performance reviewer -> traces + query plans requirements reviewer -> original user acceptance criteria verifier -> executed checks + external state Now each role has a different evidence channel.
That is much more meaningful than merely telling five copies of the same model to have different personalities.
A useful contract for adversarial roles We can represent a challenge as structured data.
from dataclasses import dataclass from enum import Enum from typing import Sequence class ChallengeType(str, Enum): ASSUMPTION = "assumption" CONTRADICTION = "contradiction" COUNTEREXAMPLE = "counterexample" MISSING_EVIDENCE = "missing_evidence" CONSTRAINT_VIOLATION = "constraint_violation" REGRESSION_RISK = "regression_risk" @dataclass(frozen=True) class Challenge: reviewer: str challenge_type: ChallengeType target_claim: str argument: str required_evidence: Sequence[str] severity: float Notice what is missing.
There is no field called:
reviewer_is_correct The reviewer produces a challenge.
The runtime still has to resolve it.
Debate should produce evidence requests A useful debate loop looks like this:
proposal ↓ challenge ↓ response ↓ unresolved claims ↓ evidence requests ↓ tools / tests / retrieval ↓ updated claims This is very different from:
agent A argues agent B argues agent A argues again agent B argues again ... The second loop can consume a large amount of compute while producing no new information.
The first loop is productive because disagreement changes the runtime’s next action.
A disagreement about database behavior should trigger a database query, schema inspection, test, trace lookup, or documentation check.
A disagreement about a repository change should trigger tests, static analysis, diff inspection, or dependency checks.
A disagreement about a research claim should trigger source retrieval.
The strongest adjudicator is often not another model.
It is new evidence.
Evidence-weighted adjudication We can model an adjudicator as a deterministic runtime over structured claims.
from dataclasses import dataclass from enum import Enum class ClaimStatus(str, Enum): SUPPORTED = "supported" REFUTED = "refuted" UNRESOLVED = "unresolved" @dataclass(frozen=True) class Evidence: source: str supports: bool strength: float fresh: bool = True @dataclass(frozen=True) class ClaimDecision: claim: str status: ClaimStatus support_score: float refute_score: float def adjudicate(claim: str, evidence: list[Evidence]) -> ClaimDecision: support = sum( e.strength for e in evidence if e.supports and e.fresh ) refute = sum( e.strength for e in evidence if not e.supports and e.fresh ) if support >= 1.0 and support > refute: status = ClaimStatus.SUPPORTED elif refute >= 1.0 and refute > support: status = ClaimStatus.REFUTED else: status = ClaimStatus.UNRESOLVED return ClaimDecision( claim=claim, status=status, support_score=support, refute_score=refute, ) The numbers here are deliberately simple.
The architectural point is more important than the scoring function:
The debate produces claims and challenges. Evidence resolves them.
What about a judge model? Sometimes external verification is incomplete.
You may need a model to compare arguments, prioritize challenges, or interpret ambiguous evidence.
That is acceptable.
But the judge should be treated as another fallible component.
The architecture should therefore record:
which arguments the judge saw which evidence it used which claims remained unresolved which model produced the judgment how often similar judgments are later contradicted A judge model should not silently convert:
uncertain disagreement into:
verified truth When strong evidence is unavailable, the correct result may be:
UNKNOWN That is not a failure of the system.
It is an accurate representation of its epistemic state.
Judge bias is a real architectural problem A judge can systematically prefer:
longer answers, more confident answers, the first answer, the last answer, familiar reasoning patterns, stylistic similarity to its own generations. So evaluation should include order randomization and argument anonymization when possible.
For example:
import random def randomized_pair(a: str, b: str) -> tuple[str, str, bool]: if random.random() < 0.5: return a, b, False return b, a, True Run the same comparison under swapped order.
If the judgment frequently flips, the judge is not stable enough to act as a strong selector.
Useful metrics include:
position sensitivity order-flip rate judge agreement with external verifier false-accept rate false-reject rate UNKNOWN rate Debate loops need hard budgets Debate has a dangerous failure mode:
critique -> rebuttal -> critique -> rebuttal -> critique -> ... There is no natural reason that another round will necessarily produce new information.
A bounded debate runtime should therefore track:
maximum rounds maximum model calls maximum tool calls maximum cost maximum wall time new-evidence rate new-challenge rate resolved-claim rate If another debate round is not generating new claims or evidence, stop.
A simple termination rule could be:
@dataclass class DebateBudget: max_rounds: int = 3 max_calls: int = 12 min_new_information: int = 1 def should_continue( round_no: int, calls: int, new_claims: int, new_evidence: int, budget: DebateBudget, ) -> bool: if round_no >= budget.max_rounds: return False if calls >= budget.max_calls: return False if new_claims + new_evidence < budget.min_new_information: return False return True The exact rule will vary by domain.
The principle is stable:
Continue debate only while it is producing information that can change the decision.
Correlated critics Suppose three reviewers all use the same model with the same prompt context.
You observe:
reviewer A: no issue reviewer B: no issue reviewer C: no issue It is tempting to interpret this as three independent confirmations.
It may actually be one failure mode sampled three times.
To estimate reviewer correlation, inject known defects.
For each defect class, record which reviewers detect it.
Example:
null bug auth bug perf bug stale state reviewer A 1 0 0 1 reviewer B 1 0 0 1 reviewer C 1 0 0 1 These reviewers are not adding much marginal coverage.
Now compare:
null bug auth bug perf bug stale state static analyzer 1 0 0 0 security model 0 1 0 0 perf profiler 0 0 1 0 state verifier 0 0 0 1 That is real functional diversity.
Measure marginal reviewer value The right question is not:
Did the reviewer find defects? It is:
Did this reviewer find verified defects that the existing review stack would otherwise miss? Useful metrics include:
verified defect precision verified defect recall unique verified defects marginal recall gain cost per unique verified defect latency per unique verified defect If reviewer C finds only defects already found by A and B, C may be unnecessary.
That is true even if C produces excellent-looking critique prose.
Debate versus self-consistency These techniques are related but solve different problems.
Self-consistency asks:
If I sample several independent trajectories, how stable is the answer? Debate asks:
Can another reasoning process find a specific defect, contradiction, or missing assumption in this proposal? Self-consistency is primarily an uncertainty signal.
Adversarial review is primarily a challenge-generation mechanism.
You can combine them, but do not collapse them into one vague “multi-agent” category.
Debate versus planner–executor–critic Planner–executor–critic separates responsibilities.
Adversarial debate adds conflict intentionally.
For example:
planner ↓ executor ↓ primary critic ↓ adversarial critic ↓ evidence acquisition ↓ verifier The adversarial critic should not merely repeat the first critic.
It should have a different contract.
Possible contracts:
find a violated requirement find a counterexample find a security boundary violation find an assumption unsupported by evidence find a regression risk find a stronger alternative explanation This makes the disagreement useful even when the adversarial critic is ultimately wrong.
Debate versus Mixture of Experts Mixture of Experts routes a task to the expert most suited to solve it.
Debate deliberately sends overlapping aspects of the same problem to multiple components so their disagreement can expose uncertainty or defects.
You might use both:
router ↓ primary implementation expert ↓ security adversary + correctness adversary ↓ verifier But the architecture should explain why each extra reviewer exists.
If the answer is merely:
more opinions should be safer that is not enough.
A complete adversarial-review skeleton Here is a compact provider-agnostic runtime.
from dataclasses import dataclass, field from typing import Protocol @dataclass class Proposal: text: str claims: list[str] @dataclass class ReviewResult: reviewer: str challenges: list[Challenge] @dataclass class VerificationResult: status: str evidence: list[Evidence] class Reviewer(Protocol): name: str def review(self, proposal: Proposal) -> ReviewResult: ... class EvidenceResolver(Protocol): def resolve(self, challenge: Challenge) -> list[Evidence]: ... @dataclass class AdversarialReviewRuntime: reviewers: list[Reviewer] resolver: EvidenceResolver max_rounds: int = 2 def run(self, proposal: Proposal) -> list[ClaimDecision]: all_challenges: list[Challenge] = [] for reviewer in self.reviewers: result = reviewer.review(proposal) all_challenges.extend(result.challenges) decisions: list[ClaimDecision] = [] for challenge in all_challenges: evidence = self.resolver.resolve(challenge) decision = adjudicate( challenge.target_claim, evidence, ) decisions.append(decision) return decisions A production version would add:
deduplication, challenge clustering, budgets, reviewer reliability estimates, state identity, evidence provenance, authorization checks, and a final goal verifier. But the basic architecture remains simple.
Coding agents Adversarial review is especially useful in coding systems because there are strong external verifiers.
Example pipeline:
implementation ↓ correctness reviewer security reviewer performance reviewer requirements reviewer ↓ tests / linters / analyzers / traces ↓ verifier The important point is that reviewers should generate checkable claims.
Weak critique:
This code may have concurrency problems. Stronger critique:
Two workers can update the same row after reading the same version. Add a concurrent test that starts both transactions before either commit. Now the challenge can be verified.
Code review For pull-request review, adversarial agents can focus on different defect classes.
For example:
reviewer A -> semantic correctness reviewer B -> authorization boundary reviewer C -> backwards compatibility reviewer D -> tests missing for changed behavior But you should measure whether each reviewer adds unique validated findings.
If four reviewers repeatedly flag the same naming issue and miss the same state-transition bug, the architecture is mostly noise.
Research agents Research systems benefit from adversarial roles because claims often depend on uncertain evidence.
Useful roles include:
claim builder source skeptic counterexample hunter methodology reviewer citation verifier But the debate should resolve through source inspection.
A reviewer saying:
I do not trust this paper is not enough.
A useful challenge says:
The conclusion depends on subgroup X, but Table 4 reports only aggregate results. Retrieve the appendix or supplementary material before accepting the claim. Now the disagreement drives evidence acquisition.
Architecture and design review Adversarial review is useful when evaluating architectural decisions.
Suppose the proposal is:
Introduce a distributed queue between service A and service B. Different reviewers might challenge:
operational complexity failure recovery semantics ordering requirements idempotency observability latency migration strategy But again, the goal is not to make the proposal survive rhetorical attack.
The goal is to expose assumptions that can be tested against workload, SLOs, existing telemetry, and operational constraints.
Incident response Incident response is an excellent fit because early hypotheses are often wrong.
A useful adversarial loop is:
primary diagnosis ↓ red-team diagnosis ↓ what observation distinguishes them? ↓ query logs / metrics / traces ↓ update incident hypothesis For example:
hypothesis A: database saturation hypothesis B: connection leak The next action should not be another debate round.
It should be the cheapest observation that discriminates between the two.
Customer support Support systems can use adversarial checking before high-impact actions.
A primary agent may recommend:
refund the transaction A policy reviewer can challenge:
Is the transaction eligible under the refund window? A fraud reviewer can challenge:
Does this request match a known abuse pattern? A customer-state verifier can query the actual order system.
The final outcome comes from policy plus source-of-truth state, not from agents voting on whether the customer “deserves” a refund.
Data and analytics agents Suppose an analytics agent concludes:
conversion dropped because mobile traffic changed. An adversarial reviewer can ask:
Did the event schema change? Did attribution logic change? Is the denominator stable? Did bot filtering change? Those are useful because they create testable alternative explanations.
The runtime can inspect schemas, event counts, deployment history, and cohort definitions.
Browser automation A browser agent may believe a checkout succeeded because the page contains the text:
Thank you An adversarial reviewer can challenge:
Was an order ID created? Did the cart clear? Did backend state change? Was payment actually authorized? The verifier should check the strongest available source of truth.
Again:
The debate identifies what could be wrong. The environment determines what actually happened.
Policy and constraint checking Adversarial review also works well for systems with hard rules.
Example:
proposal -> travel booking Reviewers can separately check:
budget constraint visa constraint arrival-time constraint company policy calendar conflict Many of these checks should be deterministic.
Do not use debate when ordinary constraint evaluation is stronger.
When debate is useful Debate tends to help when:
there are multiple plausible interpretations, early assumptions matter, different reviewers can access different evidence or tools, failures are costly enough to justify extra compute, external verification exists for at least part of the problem, disagreement can trigger a new observation or test. When debate is probably unnecessary Do not add adversarial agents merely because the task is important.
A deterministic workflow may be better when:
rules are known state is exact tests are cheap constraints are explicit there is one obvious verification path For example, if the question is whether a database migration applied successfully, query the migration table and inspect the schema.
Do not convene a panel of agents to discuss it.
The “jury of clones” failure mode One of the most misleading multi-agent patterns is:
same model same context same retrieval same tool access slightly different role prompt The system may produce superficially different prose but still share the same latent failure.
Call this the jury of clones.
A useful diagnostic is reviewer overlap:
finding overlap miss overlap source overlap strategy overlap model overlap If every dimension is high, adding another reviewer is unlikely to create much new information.
Reviewer independence is not binary Agents do not have to be fully independent to add value.
Instead, measure marginal diversity.
For reviewer r, estimate:
unique verified defects found by r -------------------------------- total verified defects found by r Also inspect which defect classes improve when r is present.
A reviewer may be highly redundant overall but uniquely valuable for security regressions.
That can still justify its cost.
Debate can reduce quality There is another failure mode worth taking seriously.
A correct agent can be talked out of a correct answer by a persuasive but wrong critic.
This is why “revision after criticism” should not be automatic.
A safer flow is:
proposal ↓ challenge ↓ evidence ↓ accept challenge / reject challenge / unresolved ↓ revise only if evidence justifies revision The same principle appeared earlier in critique-and-revision systems:
A critique is not evidence that something is wrong.
Adversarial review does not change that.
Debate should preserve minority hypotheses If four agents support A and one supports B, it may still be worth preserving B when:
B identifies a high-severity failure, B has stronger evidence, B is the only agent with relevant specialist capability, or the cost of checking B is low. So aggregation should not reduce everything to majority vote.
A useful challenge priority might be:
priority = severity × evidence_strength × checkability A low-probability catastrophic issue can deserve verification even when most agents disagree.
Multi-agent debate as active testing This leads to a more useful mental model.
Do not think:
multiple agents are having a conversation Think:
multiple hypothesis generators are proposing tests The runtime then chooses which test has the highest expected information value.
That turns debate into active testing.
This is far more powerful than endless textual argument.
Production telemetry A production adversarial system should log more than final answers.
Useful fields include:
task_id proposal_id reviewer_id reviewer_model reviewer_contract challenge_type target_claim evidence_requested evidence_retrieved challenge_status verified_defect round latency cost Aggregate metrics should include:
verified success rate false-accept rate false-reject rate unique verified defects per reviewer marginal recall gain reviewer overlap judge order-flip rate debate rounds per task new evidence per round cost per verified success cost per unique verified defect If you cannot see which reviewer actually improved outcomes, the architecture is very difficult to optimize.
Controlled experiment Compare at least these variants:
A. single agent B. single agent + one generic critic C. single agent + targeted adversarial critic D. single agent + two functionally distinct critics E. multi-round debate F. adversarial review + external evidence acquisition Measure:
verified task success false-pass rate unique verified defects calls latency cost The key question is not:
Did debate produce a more convincing answer? It is:
Did debate expose failures that the simpler system missed, and did those discoveries improve externally verified outcomes enough to justify the extra compute? Failure attribution A multi-agent debate system should let us answer exactly where it failed.
Proposal failure No correct hypothesis was generated.
Challenge failure The proposal contained a defect, but no reviewer challenged it.
Evidence failure A useful challenge existed, but the runtime did not gather the evidence needed to resolve it.
Adjudication failure The evidence existed, but the judge or policy interpreted it incorrectly.
Revision failure A valid challenge was accepted, but the revised result introduced another defect.
Verification failure The system accepted an outcome despite insufficient or stale evidence.
This decomposition is far more useful than saying:
multi-agent debate failed A practical escalation ladder Start simple.
single agent ↓ external verification ↓ targeted critic for a measured blind spot ↓ functionally distinct second critic ↓ evidence-driven adjudication ↓ bounded multi-round debate Do not start with ten personas and a judge.
Earn each role.
The deeper pattern Across this entire advanced-agent series, the same principle keeps returning.
Search does not create truth.
Consensus does not create truth.
A planner does not create truth.
A critic does not create truth.
A judge does not create truth.
A debate does not create truth.
These mechanisms help the system decide where to look next.
They allocate compute.
They surface uncertainty.
They generate hypotheses.
They identify possible defects.
They propose tests.
But when the environment can answer the question more directly, the architecture should eventually leave the model and ask the environment.
That gives us the core rule for adversarial agents:
Use debate to discover what must be checked. Use evidence to decide what is true.
Next: adaptive agents So far, most of our advanced architectures are still configured before the task starts.
We choose:
how many samples to generate, how many branches to search, which experts exist, how many critics to run, and which model tier to invoke. But tasks do not all deserve the same amount of compute.
Some are trivial.
Some become difficult only after a failed verification step.
Some reveal uncertainty halfway through execution.
The next step is to make the architecture respond to those signals.
In Advanced Agents From First Principles 08, we will build adaptive agents that allocate more reasoning, search, specialists, and stronger models only when the task actually needs them.