← Back
AWS ML Blog

Deploying Qwen3.8-2.4T-A95B on Amazon SageMaker HyperPod with vLLM

•20 min read•
#deployment#amazon#inference
✦TL;DR

The authors demonstrate how to deploy the 2.4‑trillion‑parameter Qwen3.8‑2.4T‑A95B model on Amazon SageMaker HyperPod using vLLM, achieving an OpenAI‑compatible endpoint that supports built‑in reasoning, tool‑calling, and native MTP speculative decoding. By provisioning a HyperPod cluster and applying NVFP4 quantization, the deployment reduces memory footprint while maintaining inference speed, enabling production‑grade serving of a massive open‑weight LLM. The walkthrough includes concrete steps for cluster creation, model packaging, and endpoint configuration, illustrating a practical path to scale large models on AWS infrastructure. The approach showcases how HyperPod’s GPU‑rich architecture can handle trillion‑parameter workloads with vLLM’s efficient scheduling and speculative decoding, highlighting a trade‑off between quantization precision and latency.

⚡ Key Takeaways

  • Qwen3.8‑2.4T‑A95B is a 2.4‑trillion‑parameter open‑weight model deployed on SageMaker HyperPod.
  • NVFP4 quantization is applied via vLLM to reduce memory usage while preserving inference quality.
  • The endpoint is OpenAI‑compatible and includes built‑in reasoning, tool‑

Want the full story? Read the original article.

Read on AWS ML Blog ↗

More like this

New in AWS Quick: More capable flows, an updated web experience, and agents that work together

AWS ML Blog•#agents

Three insights you may have missed from theCUBE’s coverage of the Dell Technologies New York Symposium

SiliconANGLE AI•#deployment

Welcome Inkling 2 Mini, Cutting-edge Performance for your Local Setup

Hugging Face Blog•#llm

NVIDIA and Microsoft Expand Accelerated Computing From Azure to Windows PCs

NVIDIA Blog•#compute

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING