← Back
AWS ML Blog

Use decision models with Amazon Bedrock AgentCore Evaluations

•15 min read•
#agents#bedrock#amazon
Level:Advanced
For:AI Engineers
✦TL;DR

Amazon Bedrock AgentCore Evaluations can incorporate decision models as custom code‑based evaluators, enabling fast, calibrated probability scores for fixed‑answer questions such as policy compliance or grounding. The post demonstrates wiring two decision‑model backends—Jev (a hosted service from TypeSafe) and Strands Decider 2B (an open‑source model you run yourself)—through an AWS Lambda function that translates AgentCore inputs into model requests and maps the returned probabilities back to AgentCore evaluation results. This approach lets engineers add a low‑latency screening layer before invoking the 15 built‑in evaluators, improving throughput while retaining explainability for deeper analysis.

⚡ Key Takeaways

  • AgentCore Evaluations includes **15 built‑in evaluators** and supports custom evaluators defined as either **LLM‑as‑a‑judge** or **code‑based**.
  • Custom code‑based evaluators run in **AWS Lambda** and can call external systems, including decision‑model backends.
  • The post integrates two specific decision‑model backends: **Jev** (hosted) and **Strands Decider 2B** (open‑source).
  • Decision models return a **probability distribution over predefined answers**, providing a calibrated confidence score that can be thresholded or aggregated.
  • By using decision models as a **screening layer**, you can generate a pass/fail signal for multiple metrics in a single request, reducing the load on the more expensive built‑in evaluators.
💡 Why It Matters

Engineers can now add inexpensive, high‑throughput decision‑model checks to their agent evaluation pipelines, enabling real‑time policy and grounding validation without sacrificing the deeper diagnostics offered by Bedrock’s native evaluators.

✅ Practical Steps

  1. Implement an **AWS Lambda** function that receives the AgentCore evaluator input, formats it into a request for the chosen decision model (e.g., Jev or Strands Decider 2B), and parses the returned probability distribution.
  2. Map the decision model’s probabilities to an **AgentCore evaluation result** (label + confidence) and return it to the AgentCore evaluation workflow.
  3. Deploy the Lambda as a **custom code‑based evaluator** in AgentCore Evaluations, then configure your evaluation job to invoke it alongside or before the built‑in evaluators.

Want the full story? Read the original article.

Read on AWS ML Blog ↗

More like this

Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod

AWS ML Blog•#deployment

Introducing Gemini 4 Neon

Google DeepMind•#llm

DaisyRose-27B: Pushing the Frontier of Post-Training Efficiency

Hugging Face Blog•#llm

Mathematical proof, powered by AI. And no hallucinations!

Amazon Science•#agents

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING