← Back
Towards Data Science

Keyword Search Tied an 8B Embedding Model at Picking Agent Tools

•
#agents
Level:Intermediate
For:ML Engineers
✦TL;DR

On a benchmark of 200 ToolRet questions covering up to 3,794 tools, BM25 keyword search ran roughly 40× faster than the 8 B BGE‑large embedding model on a CPU, and using full tool documentation made BM25 the top‑performing configuration. This shows that a classic lexical retriever can outpace a large embedding model for rapid tool selection in agent pipelines. Engineers can therefore favor BM25 with comprehensive docs when low latency tool lookup is critical.

⚡ Key Takeaways

  • BM25 achieved ~40× faster inference than BGE‑large on CPU for 200 ToolRet queries.
  • Using full tool documentation with BM25 yielded the best retrieval accuracy among tested setups.
  • The trade‑off favors lexical search speed over the semantic richness of an 8 B embedding model.
  • Implement BM25‑based retrieval when building agent tool‑selection components that must run on CPU.
  • Limitation: performance comparison is limited to CPU; GPU‑accelerated embedding inference was not evaluated.
💡 Why It Matters

Leveraging BM25 with full docs lets production agents select appropriate tools with minimal latency, avoiding the overhead of large embedding models on modest hardware.

✅ Practical Steps

  1. Apply BM25 keyword search over the complete tool documentation when designing your agent’s tool‑selection module.
  2. Benchmark your own CPU latency to confirm the ~40× speed advantage before considering larger embedding models.

Want the full story? Read the original article.

Read on Towards Data Science ↗

More like this

Introducing Gemini 4 Neon

Google DeepMind•#llm

DaisyRose-27B: Pushing the Frontier of Post-Training Efficiency

Hugging Face Blog•#llm

Mathematical proof, powered by AI. And no hallucinations!

Amazon Science•#agents

AI’s GPU focus meets a CPU reality check at Dell and AMD

SiliconANGLE AI•#agents

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING