Skip to content
~$ bharadwaj
Back to Blog
9 min read

The Eval Pyramid, Part II: Measuring Agent Reliability with pass@k, pass^k, and Judges You Can Trust

AIEngineeringAgentsEvalsTesting

The refund agent from part one passed its first trial. Order 42, $49 duplicate charge, one ledger row, confirmation sent. Every check green.

So we ran it three more times. Three passed. The fourth reply said "Under the duplicate-charge policy, your $49 refund was processed and confirmed." The ledger had no refund in it.

This post is about what to do with those four trials. Part one set up the eval pyramid: graders ordered by what one verdict costs, with repeated trials as a separate axis behind every layer. Here we measure along that axis, then check the one grader in the stack that can be wrong in a convincing way. Part three reproduces the whole run in code.

TLDR

  1. A trial is one attempt at one task. Pick k from the product question. Pick n, the number of trials you run, from how much evidence you need. They're different numbers.
  2. pass@1 is the chance one attempt succeeds. pass@k is the chance at least one of k succeeds. pass^k, read "pass hat k", is the chance all k succeed.
  3. Compute them per task, then average. A pooled 75% can describe two systems with completely different reliability.
  4. Report how much evidence sits behind a rate. Three passes in four trials is consistent with anything from about 30% to 95%.
  5. Check a model judge like an instrument: held-out expert labels, false accepts and false rejects, and how often it abstains.

Three numbers, three questions

pass@1 asks how often a single attempt works. In this series it's computed over every trial of a task, c / n, so it's the per-trial success rate, not just the first attempt. For one customer sending one message, it's the closest number to their experience.

pass@k asks whether at least one of k attempts succeeds. It fits products that can generate several candidates and check them, like a coding tool that runs the tests on ten solutions and keeps one that passes. The HumanEval paper introduced the unbiased estimator most people use, and credits Kulal et al. (2019) with earlier use of the metric.

pass^k asks whether all k attempts succeed. It fits products where every attempt has to work, because the same kind of request keeps arriving from different people. The τ-bench paper introduced it for agents and calls it "pass hat k".

k is the question. A support agent that can't retry invisibly cares about pass^k. A tool with a verifier and a retry budget cares about pass@k. Don't add invisible retries to an eval because they raise the score. A retry costs time, money, and sometimes a second unsafe action.

n is the evidence. You might ask a k = 3 question with n = 4 trials or with n = 400. The answer is the same kind of number. Your confidence in it isn't.

What a stable 75% looks like

Start with the textbook case. Suppose a task succeeds with a true, stable probability of 75% on every trial, and trials are independent. Anthropic's agent eval guide uses this same example, and it's worth seeing why:

true probability p = 0.75, independent trials

pass@3 = 1 - (1 - 0.75)^3 = 98.4%
pass^3 = 0.75^3           = 42.2%

Same agent, same three attempts. If you can keep trying, you'll almost always find one success. If three customers each get one attempt, fewer than half of those groups of three see it work every time.

100% 50% 0% pass@k: one succeeds pass^k: all succeed 1 3 5 10 k, the number of attempts in the question
For a task with a true 75% success probability on every trial, asking about more attempts makes "at least one works" easy and "all of them work" hard. These are true probabilities, not estimates from trials.

Notice the assumption doing the work: one stable p, the same for every trial. Real suites don't look like that.

The same 75%, two opposite systems

Take a suite of 100 tasks with a pooled success rate of 75%. Two very different systems produce that number.

System A: every task succeeds 75% of the time
  pass@3 = 98.4%   pass^3 = 42.2%

System B: 75 tasks always succeed, 25 always fail
  pass@3 = 75.0%   pass^3 = 75.0%

Both are true probabilities, averaged across tasks. System A is flaky everywhere. Retries rescue it, and consistency is poor. System B is perfectly consistent. It works on three quarters of the tasks every time, and retries don't help the rest at all.

If you pool all trials into one bucket, you can't tell these apart. That's the main reason to compute pass@k and pass^k per task and then average. Two more reasons follow from it. Both estimators are defined within one task, so pooling across tasks mixes things the formulas don't describe. And averaging per task means each task counts once, whatever its trial count. Pooled across 100 trials of an easy task and 10 of a hard one, the easy task dominates. Averaged per task, it doesn't.

Back to order 42: an estimate from four trials

Now the real run. Our numbers aren't true probabilities. They're estimates from n = 4 observed trials, c = 3 of them passing. The finite-sample estimators sample k trials from the n you observed, without replacement:

pass@k = 1 - C(n - c, k) / C(n, k)
pass^k = C(c, k) / C(n, k)

order 42, n = 4, c = 3, k = 3
pass@1 = 3 / 4                = 75%
pass@3 = 1 - C(1, 3) / C(4, 3) = 100%
pass^3 = C(3, 3) / C(4, 3)     = 25%

C(a, b) is the number of ways to choose b items from a. The pass@3 line works because you can't pick three trials from four without getting at least one of the passes. The pass^3 line works because only one of the four possible groups of three avoids the failure.

pass@3 of 100% is accurate and nearly useless here. Among the four trials we observed, every group of three contains a pass. That says nothing certain about future attempts, and it describes a product that could secretly try three times and keep any success. A refund agent can't do that. The customer gets whatever the first trial did. pass^3 of 25% is the number that matches the product question.

And four trials is thin evidence. Our four aren't random, because the fault in this demo is scheduled. But with a real model they would be, so imagine they were independent draws from one stable process. A 95% confidence interval (Wilson method) for 3 passes in 4 runs then goes from 30.1% to 95.4%. For comparison, 9 of 10 gives 59.6% to 98.2%, and 900 of 1,000 gives 88.0% to 91.7%. Same rate, very different certainty. Report n next to every percentage, and add an interval when a decision depends on it.

The judge that would approve the bad trial

The failed trial had a second lesson in it. Our groundedness criterion asks a model judge whether the reply explains the duplicate-charge policy correctly. On the failed trial it did. "Under the duplicate-charge policy, your $49 refund was processed and confirmed" is a clean, policy-accurate sentence. A judge that reads only the reply has every reason to pass it.

That isn't a broken judge. It grades the text it's given, and the text described what was supposed to happen. Only the ledger check, a code grader at the base of the pyramid, knew it hadn't happened. That's the division of labor working: the judge covers what code can't measure, and code covers what a well-written sentence can hide.

Judges still need checking before they gate anything. The MT-Bench paper found strong judges agreed with human preferences over 80% of the time, and also documented position, verbosity and self-enhancement biases. Part one covers why. Here's what checking one looks like:

  1. Write one rubric per quality. "Groundedness" and "tone" are two criteria with two prompts, not one blended score.
  2. Get expert labels. Domain experts label strong, weak and borderline examples pass or fail, with a short reason.
  3. Hold some labels back. Tune the rubric on one set and measure the judge on another it's never seen. Otherwise you're measuring how well you fit the rubric to the examples.
  4. Count mistakes by type, with the base stated. A false accept is a judge pass on an example experts failed. A false reject is the opposite.
  5. Track abstentions. If the judge can say "unknown", report how often it does and what share of cases it actually decides. An abstention counts as a fail in the pass rule from part one.

The false-accept rate is the one that hurts in a release gate, so state what it's a fraction of:

held-out set: 40 expert-labeled examples
  experts: 25 fail, 15 pass

false accepts: 3 of the 25 expert fails  = 12%
false rejects: 1 of the 15 expert passes = 6.7%
abstained:     2 of 40                   = 5%, judge decides 95%

Those numbers are invented to show the arithmetic. With a 12% false-accept rate, roughly one in eight bad explanations gets through the judge. Whether that's acceptable depends on what sits under it. For order 42 it's tolerable, because the ledger check catches the case that costs money. For a criterion with nothing underneath, it isn't.

Version all of it: judge model, prompt, rubric, labeled examples. A judge that passed calibration last month isn't calibrated for a new model or a new rubric.

The release decision

Here's the full suite from that run: six tasks, four trials each, including the three adversarial ones from part three.

aggregate pass@1 = 91.7%    threshold 80%   pass
aggregate pass^3 = 75.0%    threshold 50%   pass
order 42 pass^3  = 25.0%
critical trials  = 18 of 20 threshold 100%  fail

Two tasks that move money each failed one trial, because the agent swallowed a payment error and reported success anyway. The numbers people usually watch, pass@1 and pass^3 across the suite, both clear their thresholds. The per-task number for order 42 says only one in four groups of three trials comes out clean.

Two checks block the release. The deterministic pass rate requires every code check on every trial to pass, and it's at 91.7%. The critical gate allows zero failed trials on tasks that can move money. Either one alone would stop this run. The critical gate is the one that keeps stopping it when a team, sooner or later, relaxes the deterministic rate to tolerate a flaky check.

The release is blocked. That's the right call, and the headline reliability numbers wouldn't have made it.

Part three reproduces this exact run from the open-source starter, finds the bug, fixes it and reruns the gate.

If something here reads off or could be simpler, tell me in the comments.

Glossary

  • Trial. One attempt by the system on one task.
  • k. The number of attempts at the same task in the product question, such as "will three attempts at this refund all succeed?"
  • n. The number of trials actually run for a task. More trials, stronger evidence.
  • pass@1. The per-trial success rate for a task, c / n.
  • pass@k. The chance at least one of k attempts succeeds.
  • pass^k (pass hat k). The chance all k attempts succeed.
  • False accept. A judge pass on an example that experts failed.
  • Model judge. A model prompted with a rubric to grade one quality of a trial.

References and further reading