← Back
AWS ML Blog

Right-size generative AI endpoints with concurrency sweeps on Amazon SageMaker AI

•10 min read•
#deployment#llm#inference
Level:Advanced
For:AI Engineers
✦TL;DR

This article introduces concurrency sweeps—a systematic benchmarking method built into Amazon SageMaker AI Inference Recommendations—to right‑size generative AI endpoints by identifying the optimal instance type, instance count, and concurrency level that maximize throughput (tokens per second) while keeping latency within SLA limits. The workflow deploys the NVIDIA Nemotron‑3 Nano 30B model on an ml.g7e.2xlarge instance (NVIDIA Blackwell GPU) using the native vLLM container, runs automated sweeps via the CreateAIBenchmarkJob API, and analyzes the resulting throughput‑latency curve to determine the ideal concurrency, breaking point, and required fleet size. Engineers can thus avoid over‑provisioning idle GPUs or under‑provisioning that leads to request queuing and latency spikes.

⚡ Key Takeaways

  • Concurrency sweep measures **throughput (tokens per second)** and **latency** at incremental concurrency levels (e.g., 64 → 256 → 1,024 simultaneous requests).
  • Uses the **native vLLM container** for model deployment on SageMaker AI endpoints.
  • Trade‑off: adding concurrency improves throughput only up to the **saturation point**, after which latency degrades past the SLA threshold.
  • Integration point: invoke the sweep with the **CreateAIBenchmarkJob** API.
  • Prerequisite: an **AWS account with SageMaker AI access**, an IAM execution role with SageMaker and S3 permissions, and sufficient **service quota for ml.g7e.2xlarge** endpoints.
💡 Why It Matters

By automating capacity discovery, engineers can provision exactly the number of ml.g7e.2xlarge instances needed for peak traffic, cutting cloud spend while guaranteeing SLA‑compliant latency for generative AI services.

✅ Practical Steps

  1. Deploy NVIDIA Nemotron‑3 Nano 30B to a SageMaker AI endpoint using the **native vLLM container** on an **ml.g7e.2xlarge** instance.
  2. Run a concurrency sweep by calling **CreateAIBenchmarkJob** with your desired workload profile (input/output token counts, streaming mode).
  3. Analyze the benchmark output to locate the **optimal concurrency level**, **breaking point**, and compute the **right‑size factor** for fleet sizing.

Want the full story? Read the original article.

Read on AWS ML Blog ↗

More like this

Censored vs. Truncated Regression in Python, Explained

Towards Data Science•#llm

New in AWS Quick: More capable flows, an updated web experience, and agents that work together

AWS ML Blog•#agents

Three insights you may have missed from theCUBE’s coverage of the Dell Technologies New York Symposium

SiliconANGLE AI•#deployment

How Google Duplex evolved to also call businesses for you in Gemini

Google Research•#agents

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING