How reliable AI agents actually get built,
and why the model was never the hard part.
Lead Member of Technical Staff at Salesforce. I work on enterprise workflow systems and the tooling around agents.
Most of this deck comes from things that broke on me.
Nobody buys an engine to get to work. They buy transport they can trust.
A demo has to be right once. A product has to be right the ten-thousandth time. From what we've seen, that gap is structural rather than bad luck.
The gap wasn't intelligence. It was engineering. Most teams we've worked with aren't waiting for a smarter model. They're underinvesting in the harness.
A genius with no tools, no docs, no manager, and no review gives you confident, plausible, sometimes-wrong work.
Reliability compounds multiplicatively. So does flakiness.
Over 20 steps it's a coin flip that loses two times out of three.
Expensive, asymptotic, and never enough for a truly long task. There's a ceiling.
A checked step with a retry has effective reliability far above its raw rate. No ceiling.
“The model says it's done” is not verification. A failing test is. The compiler doesn't hallucinate, so let it be the judge.
Error: invalidNothing to recover from. The model guesses and flails.
path must be absolute, got ./foo, use /repo/fooThe model self-corrects on the next turn.
Then bound the retries. Most recoverable failures we've seen resolve in 2 or 3 attempts. Cap it, or the agent loops on burning your token budget.
Not the system prompt. Constrained decoding, allowlists, permission gates, pre/post-tool hooks. Enforced by the harness, not requested of the model.
“Put it in code” isn't abstract. Here it is in a tool you can open tonight: a rule the agent can't talk its way out of, then a door it physically can't open.
The JSON is the hook. It registers a script to run before every edit. The script can say no, and the agent can't argue.
Verification becomes a wall, not a suggestion the model can skip.
"PreToolUse": [{ "matcher": "Edit|Write", "hooks": [{ "command": "…/tdd-gate.sh" }] }]
# harness passes the tool call as JSON on stdin FILE=$(echo "$IN" | jq -r '.tool_input.file_path') if [[ "$FILE" == */src/* ]] && [[ ! -f .tests-passed ]]; then echo "Run the tests before editing $FILE" >&2 exit 2 # exit 2 = BLOCK; stderr goes back to the agent fi
No script, just config. A deny list the model can't reach past.
deny is checked first and always wins. You can't allow-list around it.
{
"permissions": {
"deny": [
"Bash(rm -rf:*)", # no mass deletes
"Edit(prod/**)", # hands off prod
"Read(.env.production)" # can't read secrets
]
}
}
# checked deny → ask → allow. deny always wins.
Runs logic, checks state, decides in the moment. “Earn the edit.” Plan, act, verify with teeth.
A flat rule that never runs code and can't be reasoned with. A confused agent still can't touch prod.
You changed the model by exactly zero. Twenty lines of the car around the engine, and you can try it this afternoon.
Over a long run the agent's worst enemy is often its own transcript. One early false “fact” gets restated for 40 steps.
Infers the column is user_email. It's email. Every SQL step perfect. Migration reports clean.
Fix → flaky test fails → revert → rewrite the identical fix. No memory of failure.
Script early-returns, exits 0. Old version keeps serving. Nobody notices for hours.
“Exit 0” is not “it worked.” Verify outcomes against intent, not return codes.
It will confidently execute the disaster.
Short leash where the agent is blind. Long rope only where a test can catch the fall.
Top: no harness. Bottom: the harness, run as a business.
Automated trading with no kill switch. The cautionary tale every risk officer already knows.
The chatbot made one up; a court ordered them to honor it. A guardrail on approved policy would have prevented it.
Two-thirds of chats, after guardrails & escalation. Then in 2025 they re-hired humans when quality slipped. The envelope, live.
Redundancy, remote operators, telemetry. The rollout waited on the harness, not a smarter vision model.
Net value = (success × tasks × value) − (error + oversight + token cost)
Reliability is the multiplier on the left and the reducer on the right.
If a human re-checks everything to find the missing 10%, that's not automation. It's a scavenger hunt.
An agent that takes the wrong action can be worse than no agent. Now someone has to notice and undo it.
Evals are regression tests for agents. Invest in them first. You can't improve what you can't measure.
You and your competitor call the same frontier weights. It's a rented input.
Tools, evals, permissions, recovery, domain context. Nobody else has yours.
Build for the model you'll have, not the one you're fighting. Delete workarounds on schedule.
Prompts copy in an afternoon. A real eval suite takes months. We think that asymmetry is where the moat sits.
Every failure your harness catches is a test your competitor hasn't written yet.
Prompts are hope. Tools are leverage. Evals are proof. MCP did for tools what HTTP did for documents. Now harnesses compose, and agents are starting to build agents.
Smarter models raise the ceiling. Better harnesses raise the floor, and the floor is where reliability lives.
The harness turns raw capability into dependable behavior.
So verification and recovery beat waiting for a smarter model.
That's where trust, safety, and differentiation tend to live.
Thank you. Now, what's your “run the tests”?