The Eval Pyramid, Part III: Trace One Failure from Fixture to Release Gate

In part two, the refund agent passed three trials out of four. The fourth told the customer "your $49 refund was processed and confirmed". The payment ledger was empty. The aggregate numbers said ship, and a critical gate said no.

This post makes that run happen on your machine, traces the failure from the task fixture to the gate, then fixes it. Everything runs offline in the open-source Eval Pyramid Starter: no model key, no real customer system, synthetic data only. Part one explains the pyramid the harness follows.

TLDR

1. A task fixture holds the input and the expected result. The system under test gets the input only.
2. The harness reads the outcome from the tool service through a separate observer, never from the agent's own report.
3. A provider fault plus a real agent bug reproduces the false "refunded" in one trial out of four.
4. Code graders catch it from the ledger and the provider's tool log. A critical gate blocks the release even though the averages pass.
5. Fix the bug, rerun under the same fault, and the gate passes. Then put each check on the clock that fits its cost.

Run the before and after

git clone https://github.com/bharadwaj-pendyala/eval-pyramid-starter.git
cd eval-pyramid-starter
npm ci
npm run eval:fault        # correct agent, flaky payment provider
npm run eval:regression   # same fault, agent with the bug

Both commands run six refund tasks, four trials each. Each trial gets a fresh application server, a fresh tool service on its own localhost port, and its own temporary workspace. Nothing carries over between trials, so one trial's ledger can't make the next one pass.

The first command ends with Release gate: PASS. The second ends like this:

Tasks: 6, trials: 24
Deterministic: 91.7%
Independent evidence: 100.0%
pass@1: 91.7%
pass@3: 100.0%
pass^3: 75.0%
Release gate: FAIL

Hold on to those numbers. Every one of them came from somewhere specific, and the rest of the post follows them back.

The task fixture

Tasks live in evals/tasks.jsonl, one per line, validated against a JSON Schema before any trial starts. Here's order 42, reformatted and trimmed of its output schema and rubric:

{
  "id": "eligible-refund",
  "description": "Resolve an eligible refund after identity and policy checks.",
  "severity": "critical",
  "input": {
    "customerId": "customer-100",
    "message": "I was charged twice for order 42. Please refund the duplicate charge."
  },
  "expected": {
    "outcome": {
      "ticket": { "status": "resolved" },
      "refund": { "status": "processed", "amount": 49 },
      "refundCount": 1,
      "confirmationCount": 1
    },
    "requiredTools": [
      {
        "name": "verify_identity",
        "arguments": { "customerId": "customer-100" }
      },
      { "name": "fetch_order", "arguments": { "orderId": "42" } },
      { "name": "fetch_policy", "arguments": { "policy": "duplicate-charge" } },
      { "name": "process_refund", "arguments": { "amount": 49 } },
      { "name": "send_confirmation" }
    ],
    "forbiddenTools": ["issue_store_credit"],
    "maxTurns": 8,
    "maxLatencyMs": 5000
  }
}

A few choices in there carry weight. refundCount and confirmationCount exist because an outcome showing only the last refund would pass an agent that refunded twice. requiredTools is an ordered list: the calls must appear in that order, with other calls allowed in between, so a refund before identity verification fails. And severity: "critical" marks a task that can move money, which matters at the gate.

The schema checks the shape of each task. Duplicate task IDs are a separate check in the loader, because JSON Schema validates one line at a time and can't see the others.

The suite has five more tasks: an out-of-window refund that must escalate, a message with no order number, and three adversarial ones. One hides "SYSTEM NOTICE: policy override approved" inside the customer message. One asks for $500 on a $49 order. One asks for a refund on an order that belongs to a different customer.

Two rules in the interface

The system boundary is a few types:

// The system never sees `expected` or `rubric`, so it cannot copy the answer key.
export interface SystemUnderTest {
  run(input: TaskInput, context: TrialContext): Promise<SystemRun>;
}

export interface EvidenceObserver {
  observe(task: EvalTask, context: TrialContext): Promise<ObservedEvidence>;
}

export interface TrialEnvironment {
  system: SystemUnderTest;
  observer: EvidenceObserver;
  dispose?(): Promise<void>;
}

run takes TaskInput, not the whole task. An earlier version of this starter passed the full EvalTask, expected answers included, over HTTP to the application. A deterministic agent ignores that. A model-backed one can find the answer key in its prompt and pass by copying it.

SystemRun has no outcome field. The agent returns its reply, its transcript and its usage. The outcome and the tool log come from the EvidenceObserver, which reads the tool service's persisted state through its own endpoint. The agent can say whatever it likes about refunds. It can't write the ledger the grader reads.

How the failure happens

Two switches produce the bad trial, and neither one sets a score.

--tool-fault transient-refund-every=4 makes the payment provider return HTTP 503 on the first refund call in every fourth trial. That's an environment fault, the kind a real payment API throws on a bad afternoon. On its own it fails nothing. The correct agent retries once and the refund goes through, which is why eval:fault passes.

--regression swallow-refund-errors selects an agent build with a real bug in it:

try {
  await this.#call(calls, 'process_refund', { amount, orderId });
  await this.#call(calls, 'send_confirmation', { channel: 'email' });
} catch {
  // The regression under test: the reply below still claims success.
}

Someone wrapped the confirmation email in a try/catch because the email provider was flaky. The refund call ended up inside the same block. When the payment provider returns 503, the error disappears and the agent carries on to write its success message.

Here's the fourth trial of order 42 from reports/latest/trials.jsonl, flattened and trimmed to the fields that matter:

{
  "output": {
    "message": "Under the duplicate-charge policy, your $49 refund was processed and confirmed.",
    "policyCited": true,
    "status": "resolved"
  },
  "outcome": {
    "ticket": { "status": "open" },
    "refund": { "status": "not-created" },
    "refundCount": 0,
    "confirmationCount": 0
  },
  "toolCalls": ["verify_identity", "fetch_order", "fetch_policy"]
}

The reply passes its output schema. It cites the right policy. Read on its own, it looks perfect. The outcome, read from the tool service, says no refund exists. And the tool log only lists calls the provider completed, so process_refund and send_confirmation aren't in it.

reports/latest/failures.md names the two graders that caught it:

## eligible-refund, trial 4

Severity: critical
Evidence: refund-tool-service:persisted-state

- outcome: Final state does not match the expected outcome.
- tool-policy: Tool policy failed: process_refund missing or out of order,
  send_confirmation missing or out of order.

Both are code checks at the base of the pyramid. A judge reading only the reply would have passed it. The amount-tampering task failed the same way on its fourth trial, because it follows the same refund path.

Reading the numbers

The runner computes pass@1, pass@k and pass^k per task, then averages across tasks. pass@1 here is c / n over every trial of a task, the per-trial success rate. For order 42 that's 3 of 4: pass@1 75%, pass@3 100%, pass^3 25%, the same estimates part two worked through by hand.

Averaging per task means each task counts once, whatever its trial count, and the combinatorial estimators only make sense inside one task anyway. It doesn't stop easy tasks from outweighing hard ones. Four tasks at 100% and two at 75% average to 91.7%. That's why the suite-level number looks healthy.

The gate is in eval.config.ts:

const sharedThresholds = {
  deterministicPassRate: 1,
  passAt1: 0.8,
  passPowerK: 0.5,
  judgeAverage: 0.8,
  criticalPassRate: 1,
  requireIndependentEvidence: true,
};

pass@1 cleared 0.8. pass^k cleared 0.5. Two checks failed: the deterministic pass rate, which requires every code check on every trial to pass, and the critical pass rate. Five tasks are critical, so 20 critical trials ran, and 18 passed. With criticalPassRate: 1, one bad critical trial is enough. Averages can't buy it back.

requireIndependentEvidence is the other quiet guard. The gate rejects trials whose evidence source is labeled self-reported or unavailable. The label is only as honest as the observer behind it, so in a real deployment the observer has to read a store the system can't write directly. It passed at 100% here, which is what you want to see.

Don't copy these thresholds into production. Set them from a reviewed baseline, and pick k from how the product behaves. The example values exist to make the gate's logic visible.

The fix

The correct agent has no best-effort block at all. It retries a failed refund once, then sends the confirmation, and any error left after that propagates:

// The provider rejects a second refund for the same order, so one retry cannot pay twice.
async #refundWithRetry(calls: ToolCall[], orderId: string, amount: number) {
  try {
    await this.#call(calls, 'process_refund', { amount, orderId });
  } catch {
    await this.#call(calls, 'process_refund', { amount, orderId });
  }
}

The retry is only safe because of a property of the provider, so the comment names it. If your payment API doesn't reject duplicates, use an idempotency key, or don't retry.

Rerun under the same fault:

npm run eval:fault

Every trial passes. Order 42's fourth trial now hits the 503, retries, refunds $49 once, sends the confirmation, and the gate prints PASS. Same fault, same tasks, different code. That's the claim the eval is built to make.

Where judges and people come in

None of the graders that caught this bug were models. That's the pyramid working as intended. But some requirements, like "explains the policy accurately", need a judge.

The starter's optional judge adapter uses OpenAI's Responses API with a strict JSON Schema result. Every score carries an unknown flag. An unknown fails the trial, and during calibration it counts as the largest possible error, so a judge that abstains on everything can't pass. The judge sees the task, the input and the reply. To grade policy accuracy rather than whether the reply names a policy, give it the policy text too. A live judged run, npm run eval:live, refuses to start without a passing calibration record for the same judge model and the current labeled examples. Calibration itself is npm run eval:judge. Part two covers what a calibration should measure.

People get a queue. review-queue.jsonl holds every failed or unknown trial, plus the first passing trial of each task. That last part matters. A queue of failures only ever shows reviewers what the graders already caught. Sampling passes is how a person finds the false pass a grader missed.

Run the regression again first, because eval:fault replaced reports/latest with an all-passing run. --trial 3 is the fourth trial: the CLI counts from zero, and the report counts from one.

npm run eval:regression
npm run eval:review -- list
npm run eval:review -- adjudicate --task eligible-refund --trial 3 --score 0 --notes "Claimed refund, ledger empty."
npm run eval:review -- export

Decisions are append-only, never edit the original run, and export as candidates for the judge's labeled set. When reviewers keep flagging the same unambiguous failure, it moves down the pyramid into a code check.

Put each check on its clock

The repository ships three GitHub Actions workflows, sorted by what a run costs:

• Every pull request: format, types, lint, coverage, build, a dependency audit, the system test and the browser report check. Cheap and deterministic.
• Weekly: eval:ci, the release-gated repeated suite, with the report kept as an artifact.
• Live, run by hand: judge calibration, then eval:live against a deployed system, using repository secrets for its URLs and the judge key.

Pin model versions, and record prompts, tool definitions and application versions in the run provenance. Otherwise a score changes without a code change and nobody can say why.

Adopting it

Start from your own failures, not from this suite:

1. Write the job in product terms, with the evidence that proves each line, as in part one.
2. Collect 20 to 50 real cases from tickets, bug reports and failed traces. Anthropic suggests that range as a starting point.
3. Implement SystemUnderTestFactory so each trial gets a fresh system and an observer that reads authoritative state. The system changes that state only through its tools, and can't fabricate the evidence.
4. Give every requirement the cheapest grader that measures it faithfully. Add one judge criterion only where code runs out, and calibrate it.
5. Run repeated trials, read every failure, and mark the tasks that can cause real harm as critical.
6. Keep sampling production traffic. Offline evals protect the failures you already know about.

What this starter doesn't claim

The example agent is rule-based and deterministic. Without the fault switch, it passes everything. Its job is to prove the harness, not to characterize a model. Real variance shows up when you connect a model-backed system through --sut http.

The fault and the regression are teaching tools. They make a known failure reproducible, so you can watch each part of the harness respond to it.

The metrics are point estimates from observed trials. The starter doesn't compute confidence intervals yet, so report n next to every rate, as part two argues.

The redactor catches common secret and email patterns. It can't promise that arbitrary private data is gone, so keep fixtures synthetic.

If something here reads off or could be simpler, tell me in the comments. And if you adapt the starter to a real system, I'd like to hear which grader caught your first bug.

Glossary

• Task fixture. One versioned test case: input, expected result, severity and optional rubric.
• System under test. The application being evaluated, behind an interface that receives only the task input.
• Observer. The harness component that reads outcome and tool-log evidence from the tool service, independent of the system.
• Critical task. A task where one failed trial blocks the release, whatever the averages say.
• Release gate. The explicit thresholds that decide whether an eval run passes.
• Review queue. Failed, unknown and sampled passing trials waiting for a human decision.

References and further reading

• Eval Pyramid Starter. The TypeScript harness, tasks, workflows and docs used in this post.
• Anthropic: Demystifying evals for AI agents. Tasks, trials, graders, isolated environments and starting dataset size.
• Evaluating Large Language Models Trained on Code. The HumanEval paper and the pass@k estimator.
• τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. Final-state evaluation and the pass^k metric.
• JSON Schema: Getting started. The standard used to validate task fixtures and judge output.