Skip to content
~$ bharadwaj
Back to Blog
9 min read

The Eval Pyramid, Part I: Testing Systems That Don't Behave the Same Way Twice

AIEngineeringAgentsEvalsTesting

A failing unit test usually points at one function. A failing agent run can point at the model, the prompt, a tool schema, a stale document, the test environment, or the grader itself. And the next run might pass.

I spent the first half of my engineering career in test automation and SDET roles. Unit, integration and end-to-end tests weren't abstract categories. They were how we kept frequent releases from turning into frequent incidents. A Deep Atlas session on evaluating LLM applications pulled me back to that foundation, with one uncomfortable change: the system under test no longer behaves the same way twice.

The testing pyramid still helps. But only if you're careful about what its layers mean.

This is part one of a three-part series, and it follows one support ticket all the way through. Part two measures how often that ticket succeeds. Part three builds the harness that runs it.

TLDR

  1. Write success in product terms, then name the evidence that proves it. "The reply says refunded" isn't evidence. The payment ledger is.
  2. A grader is whatever turns evidence into a verdict: code, a model, or a person. The eval pyramid stacks graders by what one verdict costs.
  3. Put most checks in code at the base. Use a model judge only for qualities code can't measure, and calibrate it against people.
  4. A trial is one attempt at a task. Repeating trials is a separate axis that applies to every grader, not a layer of its own.
  5. Decide what "pass" means before you count passes: every mandatory check passes, an unknown score counts as a fail, and a crashed trial stays in the denominator.

One ticket, start to finish

Here's the running case for the whole series. A customer, customer-100, writes: "I was charged twice for order 42. Please refund the duplicate charge." The order cost $49. Store policy says a duplicate charge is always refundable.

A model benchmark asks how capable a model is in general. This ticket asks something narrower. Does our system, with our tools, data and policies, do this job correctly? OpenAI draws the same line between frontier model evals and contextual evals built for one workflow. The thing under test isn't the model alone. It's the model, prompt, tools, retrieval, orchestration and environment working together.

Write the success condition in product language first:

Refund exactly $49 on order 42, once.
Verify the customer owns order 42 before any money moves.
Send a confirmation.
Tell the customer what happened and why, accurately.

Now ask, for each line, where the proof lives. The agent's reply is the obvious place to look and the worst one. An agent can write "your $49 refund was processed" after the payment call failed. The reply is a claim. The evidence is the state the payment provider recorded.

  • Refund $49 on order 42, once. Evidence: one $49 refund row in the ledger. Grader: code.
  • Verify ownership before the refund. Evidence: the provider's tool log, in call order. Grader: code.
  • Send a confirmation. Evidence: one confirmation in the ledger. Grader: code.
  • Explain the duplicate-charge policy. Evidence: the reply text and the policy document. Grader: a calibrated model judge.
  • Get the policy itself right. Evidence: the policy, its edge cases, customer complaints. Grader: people.

Two details in that list matter more than they look. The harness reads the ledger from the tool service, never from the agent's own report. And the agent never sees the expected answers. If the task fixture leaks into the prompt, the agent can pass by copying the answer key. Part three shows both rules in code.

What "pass" means

Count nothing until the pass rule is written down. Most eval confusion I've seen starts here.

For this ticket, a trial passes only when every mandatory check passes. One failed check fails the trial. A judge that returns "unknown" counts as a fail, because a missing verdict isn't a positive one. And a trial that crashes, say the payment provider times out and the harness can't finish, is a failed trial. It stays in the denominator. Dropping crashed runs from the count is the fastest way to report a reliability number nobody can reproduce.

Some tasks also carry a severity. Anything that can move money is critical here, so a single failed critical trial blocks a release no matter how good the average looks. Part two shows a run where the averages people usually watch say ship, and a customer was told "refunded" when no refund existed.

Two pyramids, two questions

Mike Cohn's testing pyramid has unit tests at the bottom, service tests in the middle and UI tests at the top. Its layers measure execution scope: how much of the system runs to produce the evidence. A unit test runs one function. An end-to-end test runs everything.

That axis still exists for LLM systems, and part three uses it. But the question that hurts most in evals is a different one. What does one verdict cost, and who gives it?

testing pyramid how much runs e2e integration unit eval pyramid who gives the verdict trial n trial 1 people model judges code checks define and audit
The testing pyramid orders checks by how much of the system runs. The eval pyramid orders graders by who gives the verdict. Layer width shows intended coverage, not a cost guarantee. The faint outlines behind it are trials. Any grader can score repeated trials, so repetition isn't a layer. The dashed arrow is people defining and auditing every layer, top to bottom.

Code checks are cheap per verdict and give the same verdict every time for the same evidence. A model judge costs a model call per verdict and can disagree with itself. A person costs the most and scales the least. So you want many verdicts from code, fewer from models, and a few from people. Same economics Cohn pointed at, different axis.

Running a trial and grading it cost different things

Here's the distinction most eval pyramids blur. A ledger check is one of the cheapest verdicts you can buy. It's a comparison. But producing the ledger state means running the whole agent against a sandboxed payment provider, which can take seconds and real model tokens.

So "deterministic" and "fast" aren't the same thing. The cost of running a trial comes from execution scope. The cost of grading it comes from the grader. A full agent run graded by a ledger comparison is expensive to run and cheap to grade. A one-line classifier output graded by a domain expert is the opposite.

A second blur hides in the same place. Generation determinism asks whether the system produces the same output twice. LLM systems often don't. Grading determinism asks whether the grader returns the same verdict for the same evidence. A ledger comparison does. A judge might not. A stochastic agent can take three different valid paths to the same ledger state, and one code check can pass all three. That's usually the arrangement you want: a flexible agent, a crisp contract.

Check the outcome first. Check the path when it's part of the job

An agent run leaves two kinds of evidence. The outcome is the final state of the environment, here the ledger. The trajectory is the record of the run: messages, tool calls and intermediate results, in order.

Grade the outcome by default. Anthropic recommends this over forcing agents down one path, because several paths are often valid. Comparing every run with one golden sequence rejects good behavior.

But a correct ledger can still hide a bad run. Imagine the agent refunds order 42 before it checks who owns it. Today the customer matched, so the ledger looks perfect. Next week someone else's order gets refunded. When order, permissions or safety are part of the requirement, check the trajectory too, and read it from the provider's tool log. The agent's account of itself is one more claim. LangChain's trajectory evaluators show the trade-offs between strict, unordered, subset and superset matching.

When code runs out

"Explain the duplicate-charge policy accurately" doesn't collapse into a string comparison. Neither does "firm without sounding hostile". These are real requirements, and this is where a model judge earns its cost.

A judge reads the task, the evidence and a rubric, then scores one quality. It handles reasonable variation better than exact matching. It also adds a second model to your system, with its own failure modes. The MT-Bench paper found strong judges reached over 80% agreement with human preferences, about the level humans reach with each other. It also documented position, verbosity and self-enhancement biases, plus limited reasoning on some questions. Treat a judge as a measurement instrument that needs calibration. Part two shows how.

People sit at the top. They write the rubric, label the examples a judge is checked against, settle ambiguous cases, and audit whether the automated verdicts still mean what the product needs. When they find a repeated failure, push it down: an unambiguous policy violation becomes a code check, a fuzzy one becomes a judge criterion with its own labeled examples.

Other eval pyramids, and where this one differs

I didn't invent the idea of mapping evals onto a testing pyramid. Several teams got there first, and they're worth reading:

  • Hamel Husain splits evals into unit tests, model and human evals, and A/B tests, run on different schedules because their costs differ. No pyramid, but the same cost logic.
  • LangWatch stacks software tests, then evals, then agent simulations at the peak.
  • Block's agent testing pyramid, by Angie Jones, orders layers by how much uncertainty you're willing to tolerate, and repeats runs inside its upper layers.

Each one folds two axes into a single stack. A simulation layer says how much runs, not who grades it. Repeating runs only in the upper layers implies a code check doesn't need repetition, but a ledger check on a stochastic agent needs repeated trials as much as a judge does. My own first draft of this post made the same move and put "repeated task trials" as a layer between code checks and judges.

Anthropic's agent eval guide already keeps the terms apart. A task is one test case, each attempt is a trial, and a grader scores some part of it. It just doesn't use a pyramid. This series keeps that separation inside the pyramid picture: graders are the layers, trials sit behind every layer, and execution scope is its own axis. People still sample trials rather than read all of them. That's a coverage choice, not a different axis.

Put each check as low as it can go

The operating rule is the old one, applied to graders. Give each requirement to the cheapest grader that still measures what the requirement means. Then repeat the trials that matter, because one passing trial is one data point.

For the order 42 ticket, a single trial passed every check. Good. Part two asks the question that matters before you ship: one pass, so how much evidence is enough?

If something here reads off or could be simpler, tell me in the comments.

Glossary

  • Task. One test case: an input plus the success criteria for it.
  • Trial. One attempt by the system on one task.
  • Grader. Code, a model or a person that turns evidence into a verdict on some part of a trial.
  • Outcome. The final state of the environment after a trial, such as a ledger row or a changed file.
  • Trajectory. The ordered record of a trial: messages, tool calls and intermediate results.
  • Model judge. A model prompted with a rubric to grade one quality of a trial.
  • Execution scope. How much of the system runs to produce the evidence, from one function to the whole product.

References and further reading