Token Pooling-Aware Training for Multi-Vector Embedding Models
Token pooling‑aware training lets multi‑vector embedding models keep higher retrieval quality while cutting document vector storage. Using a pool_factor = 4, a normally trained model keeps 94.6 % of its unpooled NanoBEIR nDCG@10, whereas pooling‑aware training raises retention to 97.8 % at the same 4× storage reduction. The method is demonstrated with a reproducible Sentence Transformers example, showing a practical path to compress retrieval systems without sacrificing effectiveness.
⚡ Key Takeaways
- Pool_factor = 4 yields a 4× reduction in document vectors while preserving 94.6 % of unpooled NanoBEIR nDCG@10 (normal training) and 97.8 % with pooling‑aware training.
- HierarchicalTokenPooling uses Ward hierarchical clustering to merge token embeddings based on minimal increase in within‑group variance.
- Trade‑off: aggressive compression reduces storage (e.g., 200 × 128‑dim vectors at 4 bits → 12.8 KB per doc) but can hurt relevance unless the encoder is trained with the pooling loss.
- Integration: wrap the pooling loss in Sentence Transformers via the provided loss wrapper class to train the encoder end‑to‑end.
- Prerequisite: requires a corpus of token‑level embeddings and the Sentence Transformers library; pooling‑aware training only benefits models that will be compressed with token pooling.
Engineers can halve or quarter the index size of large‑scale retrieval systems while maintaining near‑original relevance, directly lowering storage costs and latency for production search services.
✅ Practical Steps
- Use the Sentence Transformers loss wrapper to train your encoder with the pooling‑aware loss as shown in the article’s code snippet.
- Apply HierarchicalTokenPooling with pool_factor = 4 to your trained model’s document embeddings before indexing.
Want the full story? Read the original article.
Read on Hugging Face Blog ↗