← Back
Hugging Face Blog

Token Pooling-Aware Training for Multi-Vector Embedding Models

•11 min read•
#llm#inference
Level:Advanced
For:AI Engineers
✦TL;DR

Token pooling‑aware training lets multi‑vector embedding models keep higher retrieval quality while cutting document vector storage. Using a pool_factor = 4, a normally trained model keeps 94.6 % of its unpooled NanoBEIR nDCG@10, whereas pooling‑aware training raises retention to 97.8 % at the same 4× storage reduction. The method is demonstrated with a reproducible Sentence Transformers example, showing a practical path to compress retrieval systems without sacrificing effectiveness.

⚡ Key Takeaways

  • Pool_factor = 4 yields a 4× reduction in document vectors while preserving 94.6 % of unpooled NanoBEIR nDCG@10 (normal training) and 97.8 % with pooling‑aware training.
  • HierarchicalTokenPooling uses Ward hierarchical clustering to merge token embeddings based on minimal increase in within‑group variance.
  • Trade‑off: aggressive compression reduces storage (e.g., 200 × 128‑dim vectors at 4 bits → 12.8 KB per doc) but can hurt relevance unless the encoder is trained with the pooling loss.
  • Integration: wrap the pooling loss in Sentence Transformers via the provided loss wrapper class to train the encoder end‑to‑end.
  • Prerequisite: requires a corpus of token‑level embeddings and the Sentence Transformers library; pooling‑aware training only benefits models that will be compressed with token pooling.
💡 Why It Matters

Engineers can halve or quarter the index size of large‑scale retrieval systems while maintaining near‑original relevance, directly lowering storage costs and latency for production search services.

✅ Practical Steps

  1. Use the Sentence Transformers loss wrapper to train your encoder with the pooling‑aware loss as shown in the article’s code snippet.
  2. Apply HierarchicalTokenPooling with pool_factor = 4 to your trained model’s document embeddings before indexing.

Want the full story? Read the original article.

Read on Hugging Face Blog ↗

More like this

Introducing Gemini 4 Neon

Google DeepMind•#llm

DaisyRose-27B: Pushing the Frontier of Post-Training Efficiency

Hugging Face Blog•#llm

NVIDIA RTX Spark Compact Desktops Out Now

NVIDIA Blog•#compute

Helping AI text detectors keep up with changing conditions

MIT News AI•#llm

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING