http://programmer.ie/tags/benchmarking/ Recent content in Benchmarking on Programmer.ie: Modern AI programming Hugo en-us Sun, 09 Aug 2026 10:10:00 +0100 http://programmer.ie/post/advanced-agents-from-first-principles-12/ Sun, 09 Aug 2026 10:10:00 +0100 http://programmer.ie/post/advanced-agents-from-first-principles-12/ <h1 id="is-your-advanced-agent-actually-better-benchmark-it-under-equal-budgets">Is Your Advanced Agent Actually Better? Benchmark It Under Equal Budgets</h1> <p>You replace one model call with eight.</p> <p>Success rises from 62% to 74%.</p> <p>Great.</p> <p>Except the new system used:</p> <ul> <li>eight times the inference,</li> <li>three extra judges,</li> <li>two rounds of critique,</li> <li>a larger context,</li> <li>a stronger verifier,</li> <li>and several times the latency.</li> </ul> <p>Did the architecture improve?</p> <p>Or did you just buy more attempts?</p> <p>This is one of the easiest mistakes to make in advanced agent engineering.</p>