Measuring Agent Reliability: pass@k, pass^k, and LLM Judges

An agent passes 75 out of 100 test runs. Is it reliable?

The answer depends on the product. A coding tool that can generate ten solutions and automatically verify them may only need one good result. A customer support agent does not get ten invisible attempts with every customer. It needs to work consistently.

That difference is easy to lose inside one average pass rate.

This is part two of a three-part series on evaluating LLM applications. Part one introduces the eval pyramid: deterministic checks at the base, repeated task runs in the middle, then model and human judgment near the top. This post explains how to measure those middle and upper layers. Part three implements every layer in a runnable starter.

TLDR

1. Run important tasks several times. One successful run proves capability, not reliability.
2. Use pass@1 for the normal first-try experience.
3. Use pass@k when the real product can try several times and keep any successful result.
4. Use pass^k (read as "pass to the power k") when every run needs to succeed.
5. Use LLM judges for qualities that code cannot measure, then test those judges against human decisions.

One run is an anecdote

Suppose an agent passes 75 out of 100 independent test runs. Its observed first-try success rate is 75%. That is pass@1.

Now ask a product question. Can the system try three times and use any successful result? Or will three customers each expect it to work on their first attempt?

Those questions lead to opposite metrics.

pass@k asks whether at least one of k attempts succeeds. It is useful when the product can generate several candidates, retry safely, or verify the good result. Code generation popularized this metric through HumanEval. If a coding model produces ten solutions and one passes the tests, pass@10 records success for that problem.

pass^k asks whether all k attempts succeed. The τ-bench paper introduced this metric to show the reliability of agents across repeated interactions. A support agent that succeeds once out of ten is not reliable, even if its pass@10 looks excellent.

For a task with a stable 75% success probability, assuming each run is independent:

pass@3=1 − (1 − 0.75)3 = 98.4%
pass3=0.753 = 42.2%

Same agent. Same three runs. pass@3 says you will almost certainly find one success if you can keep trying. pass^3 says fewer than half of three-run groups will succeed every time.

This is why a report should never quote pass@k without describing the product. If users only get one attempt, pass@10 is a capability demo, not their experience. If the system really can produce ten candidates and verify them, pass@10 may be the right operational metric.

Choose the metric from the user experience

Start with pass@1. It is the easiest number to explain and usually the closest to what one user sees.

Add pass@k only when retries or candidate selection exist in the real workflow. Do not add invisible retries to the evaluation just because they improve the score. A retry also has a cost, adds latency, and can repeat an unsafe action.

Add pass^k when consistency matters across equivalent runs. It is especially useful for customer-facing agents, long workflows, and actions where a rare failure is expensive. Choose k based on the product. Ten repeated runs might be enough to expose instability during development, while a higher-risk release gate may need more evidence.

Keep the runs independent when possible. Shared caches, leftover files, rate limits, or a database changed by an earlier run can make the results depend on one another. Anthropic recommends clean, isolated environments so each trial measures the agent instead of test contamination.

The implementation math, if you need it

The 75% example above assumes one stable probability. Real eval suites contain easy and hard tasks, so calculate the metric for each task and then average across tasks.

Run each task n times. If c of those runs pass, the finite-sample estimators used by τ-bench are:

pass@k=1 − C(n − c, k) / C(n, k)
passk=C(c, k) / C(n, k)

Calculate each expression per task, then average the task-level values.

choose(a, b) means the number of ways to select b items from a. The first formula counts groups with at least one pass. The second counts groups containing only passes.

Also report how much evidence sits behind the percentage. A 90% pass rate from 10 runs means nine successes. A 90% pass rate from 1,000 runs is much stronger evidence. Confidence intervals or bootstrap intervals make that uncertainty visible.

When a pass cannot be reduced to code

Some requirements need judgment. Was the answer grounded in the source? Did it explain the caveat clearly? Was the response useful without being needlessly long?

An LLM judge can read the task, answer, source material, and a focused rubric, then return a score. This is useful when several different answers can all be correct. It is also another model that can fail.

The MT-Bench paper found that strong LLM judges could match human preference judgments well in its tested setting. It also found position and verbosity biases. A judge may prefer the first answer it sees or reward a longer response because it looks more complete.

Treat a judge like production code:

• Give it one clear criterion at a time.
• Define each score using observable evidence.
• Test it on strong, weak, and borderline examples labeled by domain experts.
• Swap answer order in pairwise comparisons and flag inconsistent decisions.
• Version the judge model, prompt, rubric, and examples.
• Keep people involved for high-risk cases and disagreements.

OpenAI's grader guidance recommends comparing a model grader with human-labeled examples and checking for grader hacking. If your system learns one judge's quirks, the score can rise while the product gets worse.

A practical build order

1. Define the job in product language

Write the task, expected result, forbidden outcomes, and acceptable variation. "Resolve an eligible refund without violating policy" is clearer than "score above 0.8 on helpfulness."

2. Start with a small set of real failures

Use support tickets, bug reports, manual test cases, and failed production traces. Anthropic suggests starting with 20 to 50 tasks. OpenAI describes reviewing 50 to 100 outputs to build an initial error taxonomy. These are starting points, not universal thresholds.

3. Use the cheapest faithful check

Use schemas for structure, code for calculations, unit tests for executable behavior, and state comparisons for completed actions. Add a model judge only when semantic judgment is part of the requirement. One task can use several checks.

4. Repeat the task and review the failures

Record the model version, prompt, tool definitions, retrieved context, cost, and sequence of actions. Group failures by cause: wrong tool, bad arguments, retrieval miss, policy violation, unsupported claim, or test-environment failure. The categories tell you what to fix.

5. Build a layered release gate

A support agent might require every schema and permission check to pass, zero forbidden actions, no decline in pass@1, an acceptable pass^3 for important workflows, and calibrated judge scores for explanation quality. A high-risk case still goes to a person.

The threshold depends on impact and reversibility. A restaurant recommendation and a bank transfer should not share a reliability bar.

Keep sampling real traffic after release. Offline evals protect the failures you already know. Production shows you what the test set missed. OpenAI calls this loop Specify, Measure, Improve.

Capability asks whether the system can succeed. Reliability asks whether users can depend on it. pass@k and pass^k separate those questions, while calibrated judges cover the parts of quality that code cannot express. Pick each metric from the way the product actually behaves, not from the number that looks best in a launch review.

Next: build the complete eval pyramid with the open-source Eval Pyramid Starter. The repository includes the finite-sample estimators above, isolated repeated trials, deterministic graders, judge calibration, release gates, reports, and a human review queue.

Glossary

• Trial. One attempt by the system on one eval task.
• pass@1. The observed chance of success on one attempt.
• pass@k. The chance that at least one of k attempts succeeds.
• pass^k. The chance that all k attempts succeed.
• LLM judge. A model prompted with a rubric to grade another system's output.
• Golden set. A curated set of examples and expert labels used to test a grader or system.
• Error taxonomy. A set of recurring failure categories found by reviewing failed runs.

References and further reading

• Anthropic: Demystifying evals for AI agents. Repeated trials, clean environments, task design, and agent grading.
• OpenAI: How evals drive the next chapter in AI for businesses. Contextual product evals and the Specify, Measure, Improve loop.
• OpenAI API: Graders. Grader types, calibration, and grader hacking.
• Evaluating Large Language Models Trained on Code. The HumanEval paper and the pass@k estimator.
• τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. Final-state evaluation and the pass^k reliability metric.
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Model judges, human agreement, and known biases.