Build the Eval Pyramid: A Starter Harness for LLM and Agent Testing

An eval methodology becomes useful when another engineer can clone it, replace one boundary, and get trustworthy evidence without rebuilding the whole stack.

That is the goal of the open-source Eval Pyramid Starter. It is a provider-neutral TypeScript harness that runs entirely offline by default. The example system is synthetic, the reports stay local, and no credential is required. You can inspect every layer before connecting a real model or tool.

This is part three of the Eval Pyramid series. Part one explains why the layers exist. Part two explains pass@k, pass^k, and judge calibration. This article connects those ideas to code.

TLDR

1. Keep one typed task and trial contract across deterministic graders, repeated runs, model judges, and human review.
2. Put machine-checkable requirements first: schema, outcome, tool policy, latency, turns, and cost.
3. Give every trial a fresh system instance and temporary workspace.
4. Calculate reliability per task, then average across tasks. Report pass@1, pass@k, and pass^k together.
5. Calibrate model judges against human labels and fail closed when their output is missing or invalid.
6. Make release gates explicit, retain failure evidence, route uncertain cases to people, and run cheap checks on every pull request while running broader evals on a schedule.

Clone the runnable reference

git clone https://github.com/bharadwaj-pendyala/eval-pyramid-starter.git
cd eval-pyramid-starter
npm ci
npm run eval:deterministic
npm run eval:repeat -- --trials 10 --k 3
npm run eval:judge
npm run eval:review

The default commands exercise three refund-support cases: an eligible refund, an out-of-window request, and a request missing its order identifier. They do not call an external model or touch a real customer system.

The generated evidence lives under reports/latest:

summary.json          aggregate metrics and release-gate checks
trials.jsonl          one complete record per trial
failures.md           failed graders grouped by task and trial
review-queue.jsonl    uncertain or failed cases for human labels
index.html            self-contained report for local inspection

Generated reports are ignored by Git. That matters because even a local eval can capture private output. The starter escapes dynamic HTML and redacts common credential fields, email addresses, bearer tokens, and OpenAI-style keys. Those controls are defense in depth. Use synthetic or properly de-identified fixtures and replace the redactor with one suited to your data.

One contract holds the pyramid together

Do not build four disconnected evaluation systems. Define the task once, capture the run once, and let each layer inspect the evidence it understands.

The starter's system boundary is small:

export interface SystemUnderTest {
  run(task: EvalTask, context: TrialContext): Promise<AgentRun>;
  dispose?(): Promise<void>;
}

export interface SystemUnderTestFactory {
  create(): Promise<SystemUnderTest>;
}

EvalTask describes the input, expected state, tool rules, budgets, and optional rubric. AgentRun records the output, final state, observable transcript, tool calls, and usage metrics. TrialContext supplies the run identifier, trial number, and an isolated workspace.

This boundary is intentionally provider-neutral. Your adapter can call a hosted model, a local model, an HTTP service, a workflow engine, or the same application binary users call. The graders do not need to know.

Layer one: make the contract executable

The example task file is JSONL, one versionable task per line. Reformatted for readability, one case looks like this:

{
  "id": "eligible-refund",
  "input": {
    "customerId": "customer-100",
    "message": "I was charged twice for order 42."
  },
  "expected": {
    "outcome": {
      "ticket": { "status": "resolved" },
      "refund": { "status": "processed", "amount": 49 }
    },
    "requiredTools": [
      { "name": "verify_identity" },
      { "name": "fetch_policy" },
      { "name": "process_refund", "arguments": { "amount": 49 } },
      { "name": "send_confirmation" }
    ],
    "forbiddenTools": ["issue_store_credit"],
    "maxTurns": 8,
    "maxCostUsd": 0.05,
    "maxLatencyMs": 5000
  }
}

The checked-in JSON Schema rejects malformed tasks and duplicate identifiers before a trial starts. After the run, deterministic graders check the output schema, expected final state, required and forbidden tools, turn count, cost, and latency.

These checks answer concrete questions. Did the refund reach the correct state? Did identity verification happen? Did the agent avoid an unapproved action? They are cheap, stable, and easy to debug, so they belong on every pull request.

Avoid grading private chain-of-thought. The harness records observable outputs, actions, and state transitions. Those are the artifacts a team can retain, inspect, and govern.

Layer two: repeat isolated trials

A fresh adapter instance and temporary workspace are created for every trial. Cleanup runs even after a failure. That prevents one test's files or in-memory client state from leaking into the next.

Your adapter still owns isolation outside the process. If it writes to a database, use a disposable tenant, namespace, transaction, or mocked tool layer. If it calls a model or tool, configure request and operation timeouts. The harness grades reported latency but cannot safely terminate arbitrary external work on your behalf.

For each task, let n be the number of trials and c the number that passed. The starter implements the finite-sample estimators from part two:

pass@k=1 − C(n − c, k) / C(n, k)
passk=C(c, k) / C(n, k)

The runner calculates these per task before averaging. That keeps a large easy task group from drowning out a small difficult one.

Report the three perspectives together:

• pass@1 is the observed first-attempt experience.
• pass@k measures whether at least one of k sampled attempts succeeds.
• pass^k measures whether all k sampled attempts succeed.

These are sample summaries, not confidence intervals. A high-impact release gate may also need confidence bounds, failure severity, and task-specific thresholds.

Layer three: calibrate judgment before trusting it

Some requirements are semantic. The explanation may need to cite the relevant policy, include the amount, and make the next step clear without matching one exact sentence.

The starter defines a JudgeProvider interface and includes two implementations. The fake signal judge keeps the default workflow offline and demonstrates calibration mechanics. The optional OpenAI adapter uses the Responses API with a strict JSON Schema result. Neither adapter is part of the system-under-test contract.

Every judge returns four fields:

export interface JudgeScore {
  score: number;
  confidence: number;
  rationale: string;
  unknown: boolean;
}

unknown is important. Insufficient evidence should not become a confident zero or a guessed pass. Invalid structured output throws an error, which the trial runner records as a failure.

The command npm run eval:judge compares the offline judge with a human-scored golden set of strong, weak, and borderline examples. A real judge needs a larger domain-specific set, labels from people who understand the product, and periodic recalibration after any model, prompt, or rubric change.

A judge can also be prompt-injected by the output it grades. Keep the rubric focused, delimit untrusted material, require structured output, and retain human review for disagreement and high-impact cases. Never make a model judge the only control over an irreversible action.

The human layer is a queue, not a ceremony

Failed trials and grades marked unknown flow into review-queue.jsonl. A reviewer can add a human score and notes without reconstructing the run from CI logs.

This creates a useful loop:

1. Review the failed or uncertain evidence.
2. Decide whether the system, task, deterministic grader, or model judge was wrong.
3. Turn repeated unambiguous failures into deterministic checks.
4. Add ambiguous examples and human labels to the judge's golden set.
5. Version the task, grader, prompt, model, and threshold change together.

The queue is intentionally a file in this starter. A team can later route the same typed records into an annotation tool, issue tracker, or internal review application without changing how trials are graded.

Release gates should fail for reasons you can inspect

eval.config.ts makes the example thresholds explicit:

thresholds: {
  deterministicPassRate: 1,
  passAt1: 0.8,
  passPowerK: 0.5,
  judgeAverage: 0.8,
}

npm run eval:ci exits unsuccessfully when the release gate misses one of them. The JSON summary records every check with its actual value, required value, and pass status.

Do not copy these numbers into production. Establish a reviewed baseline, choose k from actual product behavior, and set thresholds from impact and reversibility. Add zero-tolerance gates for prohibited actions. Averages should never excuse one severe policy or safety failure.

Put each layer on the right clock

The repository includes two GitHub Actions workflows with read-only repository permissions.

The pull-request workflow installs from the lockfile, checks formatting and strict types, runs lint and coverage, builds the project, audits production dependencies, generates a report, and checks it in desktop and mobile Chromium.

The scheduled workflow runs the release-gated repeated suite each week and preserves the evidence as a short-lived artifact. A production program usually adds three clocks:

• Pull request: fast deterministic regressions and a small smoke sample.
• Scheduled: broader repeated trials and judge calibration, often against several model or prompt versions.
• Production: sampled, privacy-reviewed outcomes and incidents that reveal cases the offline set missed.

Pin models and record prompts, tools, retrieval inputs, and application versions when you need comparisons across time. Otherwise a score change can arrive without a code change and leave no useful explanation.

Replace one boundary, then grow from failures

The practical migration path is short:

1. Fork or generate a repository from the starter.
2. Replace the synthetic fixtures with representative, safe cases from your product's error taxonomy.
3. Implement SystemUnderTestFactory so each trial gets a fresh client and disposable resources.
4. Return observable outcomes, tool calls, costs, latency, and turn counts from the adapter.
5. Add deterministic checks for every requirement code can express faithfully.
6. Add one focused judge criterion only where rules run out, then calibrate it against human labels.
7. Run a baseline, inspect every failure, and set reviewed release thresholds.

Start small enough that a person can read the whole task set. Twenty well-chosen cases with failures you understand are more useful than thousands of synthetic prompts nobody reviews. Add new cases when production, red teaming, or manual testing finds a gap.

What this starter does not claim

The offline adapter is deterministic unless you ask it to inject a known failure. It proves the harness works. It does not characterize a real model.

The fake judge matches explicit text signals. It demonstrates the calibration contract. It is not a semantic evaluator.

The finite reliability metrics summarize observed samples. They do not prove independence, causality, or statistical certainty.

The redactor catches common patterns. It cannot guarantee arbitrary private data is gone.

The harness is a foundation, not a compliance system. High-impact domains still need threat modeling, least-privilege tool access, domain experts, incident review, and controls outside the eval process.

Those limitations are documented in the repository because a starter should teach its boundaries as clearly as its features.

The implementation is part of the argument

An eval program earns trust when its tasks are readable, its metrics match the product, its graders are tested, its failures retain evidence, and its release rules are visible in code.

The Eval Pyramid Starter gives you that base without choosing your model provider or annotation platform. Clone it, replace the adapter, and let real failures tell you which layer to strengthen next.

Glossary

• System under test. The application or workflow being evaluated behind a provider-neutral adapter.
• Trial context. The run identifier, trial number, and isolated workspace supplied to one attempt.
• Release gate. Explicit metric thresholds that determine whether an eval command succeeds.
• Judge calibration. Comparison of model-judge scores with trusted human labels.
• Review queue. Failed or uncertain trials retained for a human decision.
• Golden set. Versioned examples with trusted labels used to test a judge.

References and further reading

• Eval Pyramid Starter. The complete TypeScript implementation, tests, workflows, security notes, and adaptation guide used in this article.
• Anthropic: Demystifying evals for AI agents. Task design, repeated trials, isolated environments, graders, and agent-eval practices.
• OpenAI API: Graders. Grader types, human-label comparison, and grader-hacking risks.
• JSON Schema: Getting started. The validation standard used for versioned task and structured judge contracts.
• Vitest: Testing in practice. Test organization and confidence-focused testing guidance.
• Playwright: Fixtures. Isolation and reusable setup for browser-level checks.
• Evaluating Large Language Models Trained on Code. The HumanEval paper and finite-sample pass@k estimator.
• τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. Final-state evaluation and the pass^k reliability metric.