Keyword Search Tied an 8B Embedding Model at Picking Agent Tools
On a benchmark of 200 ToolRet questions covering up to 3,794 tools, BM25 keyword search ran roughly 40× faster than the 8 B BGE‑large embedding model on a CPU, and using full tool documentation made BM25 the top‑performing configuration. This shows that a classic lexical retriever can outpace a large embedding model for rapid tool selection in agent pipelines. Engineers can therefore favor BM25 with comprehensive docs when low latency tool lookup is critical.
⚡ Key Takeaways
- BM25 achieved ~40× faster inference than BGE‑large on CPU for 200 ToolRet queries.
- Using full tool documentation with BM25 yielded the best retrieval accuracy among tested setups.
- The trade‑off favors lexical search speed over the semantic richness of an 8 B embedding model.
- Implement BM25‑based retrieval when building agent tool‑selection components that must run on CPU.
- Limitation: performance comparison is limited to CPU; GPU‑accelerated embedding inference was not evaluated.
Leveraging BM25 with full docs lets production agents select appropriate tools with minimal latency, avoiding the overhead of large embedding models on modest hardware.
✅ Practical Steps
- Apply BM25 keyword search over the complete tool documentation when designing your agent’s tool‑selection module.
- Benchmark your own CPU latency to confirm the ~40× speed advantage before considering larger embedding models.
Want the full story? Read the original article.
Read on Towards Data Science ↗