Clairic Hit 98% Accuracy on 72.8% of Questions via Confidence Scores
Cloudflare’s System One model achieves 98% accuracy on 72.8% of questions by applying a 0.9 confidence cutoff and delegating lower‑confidence queries to a larger, slower LLM such as Luna. This routing strategy lets the fast model handle easy cases while preserving overall quality through the fallback model. Engineers can leverage confidence‑based triage to balance latency and accuracy in multi‑model pipelines.
⚡ Key Takeaways
- 98% accuracy attained on 72.8% of questions using a 0.9 confidence threshold.
- System One acts as the front‑line model, routing sub‑threshold queries to Luna.
- Trade‑off: high‑confidence queries are answered quickly, but lower‑confidence cases incur higher latency from the larger model.
- Integration requires exposing a confidence score API from the front‑line model to decide routing.
- Prerequisite: the front‑line model must provide reliable confidence estimates; otherwise routing decisions may degrade performance.
By automatically routing only uncertain queries to a heavyweight model, teams can dramatically cut average response time while still meeting strict accuracy targets in production AI services.
✅ Practical Steps
- Deploy System One with confidence scoring enabled and set a cutoff of 0.9 to filter easy queries.
- Configure a routing layer that forwards any query with confidence < 0.9 to the Luna model for processing.
- Monitor the proportion of queries handled by each model to ensure the 72.8% coverage and 98% accuracy targets are met.
Want the full story? Read the original article.
Read on Towards Data Science ↗