← Back
Hugging Face Blog

The Agent Said It Was Done. The Database Disagreed.

•13 min read•
#agents#mcp
Level:Advanced
For:AI Engineers
✦TL;DR

Microsoft’s ThinkingBox benchmark evaluates AI agents by grading the terminal database state and side‑effects they leave behind, running each of 507 stateful business workflows 20 times across 12 LLM models. In a 121,680‑trial ablation, 79,853 attempts failed the executable checks, with 67.24 % terminating cleanly yet still producing wrong field values (77.61 %), unintended extra effects (43.30 %), or missing required effects (25.36 %). The results expose a large consistency gap between tool‑call correctness and actual backend state, urging engineers to validate agents against real data stores rather than only LLM outputs.

⚡ Key Takeaways

  • 507 workflows × 20 runs each were evaluated, totaling 121,680 valid trials.
  • The benchmark runs agents in isolated MCP tool sessions and checks terminal backend state via executable checks.
  • 67.24 % of failed attempts still terminated cleanly, but 77.61 % had wrong field values, highlighting a hidden failure mode.
  • The benchmark is available on Hugging Face and can be executed through the OpenEnv framework.
  • Executable checks focus on a single field (e.g., ticket status) to determine pass/fail, as illustrated in `sandbox_external_retail_group1.py:test_case_ST003_006`.
💡 Why It Matters

Engineers can no longer rely on surface‑level tool‑call success; validating the actual database state is essential for trustworthy agents in production systems.

✅ Practical Steps

  1. Clone the ThinkingBox repository from Hugging Face and install the OpenEnv environment as described in the paper.
  2. Run the provided benchmark script `sandbox_external_retail_group1.py:test_case_ST003_006` to reproduce the single‑field executable check.
  3. Analyze the pass@1 and 20/20 scores to assess your agent’s consistency across repeated runs.

Want the full story? Read the original article.

Read on Hugging Face Blog ↗

More like this

Introducing Gemini 4 Neon

Google DeepMind•#llm

DaisyRose-27B: Pushing the Frontier of Post-Training Efficiency

Hugging Face Blog•#llm

Mathematical proof, powered by AI. And no hallucinations!

Amazon Science•#agents

AI’s GPU focus meets a CPU reality check at Dell and AMD

SiliconANGLE AI•#agents

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING