AWS User Group DFW

Beyond LLMs

Decision models for smarter AI agents

Bharadwaj Pendyala

Frisco, TX · Nov 19, 2026

Hands up

Who has an agent that calls an LLM just to pick from a list?

Which team? Is it urgent? Yes or no? And it waits seconds for every answer.

One agent run

Color every step: decide or generate

codeDo-not-contact?
decideWhat do they want?
decideWorth an email?
codeLook up account
generateWrite the email
decidePromises anything?
decideSend or ask the AE?

The rule: if you could write down every possible answer today, it’s a decision. Seven steps here, and only one needs prose.

Who's talking

Bharadwaj
Pendyala

Tech Lead at Salesforce. I work on enterprise workflow systems and the tooling around agents.

Slides and demo code are linked at the end.

The tax

What a decision costs on an LLM

~10sto sort one support ticket with three list questions. Claude Haiku via claude -p on my laptop, CLI startup included.
Strings
reply.match(/\{[\s\S]*\}/)

Find the JSON in the prose. Hope the fields are right.

Self-reported confidence

“97% sure.” The model writes that number as text. Nothing measures it.

The idea

If the answer comes from a short list, don’t pay for prose.

Two kinds of thinking

System 1 catches the ball. System 2 does your taxes.

System 2: LLMSystem 1: decision model
Good atPlanning, writing, new problemsPicking from a short list
OutputText, one token at a timeAnswers from your list, all at once
ConfidenceA number it writesProbabilities it’s trained to get right
Your codeParsesBranches

From Daniel Kahneman, Thinking, Fast and Slow (2011). An analogy, not a spec.

Mechanically

A decision model is a function call

inState
+
inQuestions + allowed answers
→
modelOne request
→
outOne answer each
+
outProbabilities

Questions run in parallel. Answers always come from your list. That guarantees the shape, not the truth.

The building blocks

Four question types

Choice

One of a list

Which team? What does the lead want?

Score

Where on a scale

How urgent, from 0 to 2? Returns 1.7, not just “high”.

Yes or no

How likely true

Does the draft promise a discount? Jev: noul. OpenAI: predicate.

Box

Where in an image

Jev only, experimental.

Example 1

Jev, from TypeSafe

  • Trained for calibrated probabilities (they call it RLCD).
  • Choice, score, noul, box.
  • Vendor claim: 70 to 500 ms end to end.

We’ll measure it live instead of trusting the slide.

POST api.typesafe.ai/v1/systemone2-jev.ts
{
  model: 'jev-latest',
  state: { ticket },
  questions: {
    department: {
      type: 'choice',
      instructions: 'Which team should handle this?',
      criteria: DEPARTMENTS,
    },
    urgency: { type: 'score', ... },
    wantsRefund: { type: 'noul', ... },
  },
}

RLCD: reinforcement learning for calibrated decisions

Example 2

OpenAI Decisions API

  • Generally available, one model: gpt-6-luna.
  • Choice, score, predicate.
  • OpenAI: about 10x faster than the Responses API for these tasks.

Two vendors shipped nearly the same shape.

POST api.openai.com/v1/decisionsdocs example
{
  model: 'gpt-6-luna',
  input: ticket,
  questions: [
    {
      type: 'choice',
      name: 'department',
      instructions: 'Which team should handle this?',
      choices: [...],
    },
    { type: 'score', name: 'urgency', ... },
    { type: 'predicate', name: 'wantsRefund', ... },
  ],
}
The architecture

LLM plans. Decision model branches. Code sets the line.

system 2LLM plans the steps
→
system 1Decision model at each branch
→
codeConfidence check
Sure enough

Act on it.

Not sure

Ask the LLM to think it through.

Risky or lost

A person decides.

Tonight’s agent

The agent writes the email. Jev decides if it should.

A booth follow-up agent for an account executive, built on AWS.

On AWS

One state machine. Two questions to Jev.

code · DynamoDBDo-not-contact?
decide · JevWhat do they want?
generate · StrandsAgent writes the draft
decide · JevPromises anything?
code · LambdaSend or ask?
Sure and clean

SES sends it.

Not sure

Step Functions waits. The AE approves from email.

Unclear or opted out

No email.

Architecture

How a booth lead flows through AWS

Booth lead architecture on AWSAWS us-east-1THIRD-PARTY APISINBOXESLaptopmake runParameter StoreCRM tableStep FunctionsLambdaApprove LambdaRuns tableSESBedrockoptionalreceiptsTypeSafe JevAnthropic APIClaudeCustomerAE inbox12345678a8b8a8b99
The stack

Each AWS piece has one job

Step Functions

Runs the steps in order and waits for the AE. The console shows each lead’s path.

Lambda

One function runs every step. The thresholds live here, in code.

Strands + a model

The agent loop and two tools. Bedrock or the Anthropic API, picked by one setting.

DynamoDB

A fake CRM, and a receipt for every run.

SES

Sends the email. In sandbox mode, only to your verified inbox.

Parameter Store

Holds the Jev and Anthropic API keys as SecureStrings.

One SAM template deploys all of it.

The pattern

Decide before you generate. Check after.

Choice · before

What do they want?

Meeting, pricing, info, or unclear. One cheap call picks the path before the agent runs.

Yes or no · after

Does the draft promise anything?

A discount, a date, a feature. The fast model checks the slow model’s work.

The do-not-contact check never reaches Jev. It’s a rule, so it’s an if.

Code owns the line

The whole policy is one function

Step Functions branches on the label. No numbers live in the state machine.

beyond-llms-aws/src/policy.pypolicy
SURE_ENOUGH = 0.8
LIKELY_NOTHING = 0.3
LIKELY_PROMISE = 0.5

def route(receipt, sure_enough=SURE_ENOUGH):
    intent = receipt['intent']
    unclear = intent['probabilities']['unclear']
    if intent['choice'] == 'unclear' \
            or unclear >= LIKELY_NOTHING:
        return 'skip'
    if 'promises' not in receipt:
        return 'draft'
    if intent['confidence'] < sure_enough \
            or receipt['promises'] >= LIKELY_PROMISE:
        return 'approve'
    return 'send'
Live

Demo: five booth leads

One state machine. Two Jev calls. One AE on a phone.

Replay

What you just saw

LeadThe noteWhat decided itPath
L-1Pricing before a Q1 budget reviewJev: clearly pricing, clean emailSent
L-2Beat a vendor’s price, SSO date, maybe a callJev: pricing or a call?AE approved
L-3Grabbed a sticker and a t-shirtJev: maybe nothing to answerNo email
L-4Wants a call, opted out in the CRMDo-not-contact rule, no JevBlocked
L-5Call with an SE about Step FunctionsJev: clearly a meeting, clean emailSent

The agent ran three times. Jev ran seven times. A person looked once.

Your turn

Pick a new threshold. Nothing calls a model.

Every receipt is in DynamoDB. Rerouting reruns the policy function and nothing else.

terminalNov 19 run
$ make reroute T=0.45
Same Jev answers, no new model calls. Threshold 0.8 → 0.45

L-1  pricing  100%  promises  6%     send → send
L-2  pricing   49%  promises 15%  approve → send  ← changed
L-3  info      48%                   skip → skip
L-4  -                           blocked
L-5  meeting  100%  promises 13%     send → send
Know the limits

When not to use a decision model

You need words

Replies, code, summaries. Use the LLM.

You need several hops

Look up, think, look up again. That’s System 2.

It’s a hard rule

“Don’t email people who opted out” is an if. Write it.

A fixed list guarantees the shape, not the truth.

Trust, but count

Check the calibration on your own data

Model saidAnswers in bucketActually rightVerdict
90 to 100%5046 (92%)Calibrated
70 to 90%3024 (80%)Calibrated
Below 70%209 (45%)Send to a person

Example numbers. A hundred labeled examples and an afternoon gets you yours.

Take home

Three things

  1. Color your agent’s steps with make color. Count the decisions.
  2. Move them to a decision model. Let code own the threshold.
  3. Measure calibration on your data before you trust anyone’s.

My bet: in a year, agent harnesses ship a decision model next to the LLM by default.

Questions?

Thank you.

Arrow keys to navigate / N notes / F fullscreen