← Back
AWS ML Blog

Benchmarking small LLM inference on SageMaker AI: G7 vs G5 and G6

•15 min read•
#deployment#llm#amazon
Level:Advanced
For:AI Engineers
✦TL;DR

This article benchmarks two 30B Mixture‑of‑Experts models on Amazon SageMaker AI inference across G5, G6, and the newer G7 GPU instance families, showing that the Blackwell‑based G7 (RTX PRO 4500) can achieve superior throughput, latency, and cost‑per‑token despite using only two GPUs and 64 GB of total GPU memory. The Qwen3‑Coder‑30B model is evaluated with the DJL Large Model Inference (LMI) container on ml.g5.12xlarge (A10G), ml.g6.12xlarge (L4), and ml.g7.12xlarge (RTX PRO 4500), while the NVIDIA Nemotron‑3‑Nano‑30B‑A3B‑NVFP4 model is assessed via SageMaker Generative AI Inference Recommendations with vLLM on comparable configurations. Engineers can leverage SageMaker’s automated recommendation feature to select the most price‑performance‑optimal instance for their LLM workloads.

⚡ Key Takeaways

  • Benchmark includes Qwen3‑Coder‑30B and NVIDIA Nemotron‑3‑Nano‑30B‑A3B‑NVFP4 models on 12 xlarge SageMaker instances.
  • G7 instances use two RTX PRO 4500 Blackwell GPUs with 64 GB total memory, half the accelerators of G5/G6 which use four GPUs with 96 GB.
  • SageMaker AI Generative AI Inference Recommendations automates configuration selection using real‑world GPU performance data.
  • Direct benchmarking uses the DJL Large Model Inference (LMI) container; automated optimization uses vLLM with the recommendation service.
  • G7 can deliver better price‑performance even with fewer GPUs and less memory, depending on model architecture and quantization.

🔧 Tools & Libraries

Amazon SageMaker AI Generative AI Inference RecommendationsDJL Large Model Inference (LMI) containervLLM
💡 Why It Matters

By adopting SageMaker’s recommendation engine and the G7 Blackwell instances, engineers can cut inference latency and cost per token for large‑scale LLM services without over‑provisioning GPU resources.

✅ Practical Steps

  1. Deploy Qwen3‑Coder‑30B to SageMaker AI endpoints on ml.g5.12xlarge, ml.g6.12xlarge, and ml.g7.12xlarge using the DJL Large Model Inference (LMI) container and run latency/throughput benchmarks.
  2. Use SageMaker AI Generative AI Inference Recommendations with vLLM to evaluate Nemotron‑3‑Nano‑30B‑A3B‑NVFP4 across G6 (L4), G6e (L40S), and G7 (RTX PRO 4500) configurations and select the optimal price‑performance instance.

Want the full story? Read the original article.

Read on AWS ML Blog ↗

More like this

Censored vs. Truncated Regression in Python, Explained

Towards Data Science•#llm

New in AWS Quick: More capable flows, an updated web experience, and agents that work together

AWS ML Blog•#agents

Three insights you may have missed from theCUBE’s coverage of the Dell Technologies New York Symposium

SiliconANGLE AI•#deployment

How Google Duplex evolved to also call businesses for you in Gemini

Google Research•#agents

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING