← Back
AWS ML Blog

Exa boosts embedding throughput by 50% to index billions of documents on AWS AI chips

•8 min read•
#compute#deployment#inference
Level:Advanced
For:AI Engineers
✦TL;DR

Exa increased embedding throughput by roughly 1.5× (≈ 50 % boost) on AWS Neuron hardware by applying a series of optimizations—including weight‑layout alignment and FP16 data‑parallel pipelines—across Inferentia2 xlarge and Trainium2 instances. The effort was driven by a backfill of 30 billion high‑priority documents that temporarily doubled the embedding load, forcing a re‑evaluation of hardware and software choices. These gains translate into lower per‑document embedding cost and faster ingestion for billion‑scale semantic search indexes.

⚡ Key Takeaways

  • 30 billion high‑priority documents were re‑embedded, doubling the inference load.
  • Weight‑layout optimization eliminated unnecessary transposes, contributing to the throughput gain.
  • Throughput improved roughly 1.5× (≈ 50 % increase) over the naive compile baseline.
  • The workload runs on a single NeuronCore per pod, packed onto larger Inferentia2 xlarge (inf2.xlarge) and Trainium2 instances for container homogeneity.
  • Optimization required using the AWS Neuron SDK and collaboration with the AWS Trainium team.
💡 Why It Matters

Engineers can achieve significant cost savings and faster data onboarding at web‑scale by applying similar Neuron‑SDK‑driven optimizations, enabling more responsive semantic search and agent grounding.

✅ Practical Steps

  1. Apply weight‑layout optimization to your model’s parameters to remove unnecessary transposes when compiling for Neuron hardware.

Want the full story? Read the original article.

Read on AWS ML Blog ↗

More like this

New in AWS Quick: More capable flows, an updated web experience, and agents that work together

AWS ML Blog•#agents

Three insights you may have missed from theCUBE’s coverage of the Dell Technologies New York Symposium

SiliconANGLE AI•#deployment

Welcome Inkling 2 Mini, Cutting-edge Performance for your Local Setup

Hugging Face Blog•#llm

NVIDIA and Microsoft Expand Accelerated Computing From Azure to Windows PCs

NVIDIA Blog•#compute

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING