← Back
Towards Data Science

Clairic Hit 98% Accuracy on 72.8% of Questions via Confidence Scores

•
#llm
Level:Intermediate
For:AI Engineers
✦TL;DR

Cloudflare’s System One model achieves 98% accuracy on 72.8% of questions by applying a 0.9 confidence cutoff and delegating lower‑confidence queries to a larger, slower LLM such as Luna. This routing strategy lets the fast model handle easy cases while preserving overall quality through the fallback model. Engineers can leverage confidence‑based triage to balance latency and accuracy in multi‑model pipelines.

⚡ Key Takeaways

  • 98% accuracy attained on 72.8% of questions using a 0.9 confidence threshold.
  • System One acts as the front‑line model, routing sub‑threshold queries to Luna.
  • Trade‑off: high‑confidence queries are answered quickly, but lower‑confidence cases incur higher latency from the larger model.
  • Integration requires exposing a confidence score API from the front‑line model to decide routing.
  • Prerequisite: the front‑line model must provide reliable confidence estimates; otherwise routing decisions may degrade performance.
💡 Why It Matters

By automatically routing only uncertain queries to a heavyweight model, teams can dramatically cut average response time while still meeting strict accuracy targets in production AI services.

✅ Practical Steps

  1. Deploy System One with confidence scoring enabled and set a cutoff of 0.9 to filter easy queries.
  2. Configure a routing layer that forwards any query with confidence < 0.9 to the Luna model for processing.
  3. Monitor the proportion of queries handled by each model to ensure the 72.8% coverage and 98% accuracy targets are met.

Want the full story? Read the original article.

Read on Towards Data Science ↗

More like this

Censored vs. Truncated Regression in Python, Explained

Towards Data Science•#llm

How Google Duplex evolved to also call businesses for you in Gemini

Google Research•#agents

Q&A: Enabling more human-like interactions and understanding in multimodal foundation models

MIT News AI•#llm

Welcome Inkling 2 Mini, Cutting-edge Performance for your Local Setup

Hugging Face Blog•#llm

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING