Right-size generative AI endpoints with concurrency sweeps on Amazon SageMaker AI
This article introduces concurrency sweeps—a systematic benchmarking method built into Amazon SageMaker AI Inference Recommendations—to right‑size generative AI endpoints by identifying the optimal instance type, instance count, and concurrency level that maximize throughput (tokens per second) while keeping latency within SLA limits. The workflow deploys the NVIDIA Nemotron‑3 Nano 30B model on an ml.g7e.2xlarge instance (NVIDIA Blackwell GPU) using the native vLLM container, runs automated sweeps via the CreateAIBenchmarkJob API, and analyzes the resulting throughput‑latency curve to determine the ideal concurrency, breaking point, and required fleet size. Engineers can thus avoid over‑provisioning idle GPUs or under‑provisioning that leads to request queuing and latency spikes.
⚡ Key Takeaways
- Concurrency sweep measures **throughput (tokens per second)** and **latency** at incremental concurrency levels (e.g., 64 → 256 → 1,024 simultaneous requests).
- Uses the **native vLLM container** for model deployment on SageMaker AI endpoints.
- Trade‑off: adding concurrency improves throughput only up to the **saturation point**, after which latency degrades past the SLA threshold.
- Integration point: invoke the sweep with the **CreateAIBenchmarkJob** API.
- Prerequisite: an **AWS account with SageMaker AI access**, an IAM execution role with SageMaker and S3 permissions, and sufficient **service quota for ml.g7e.2xlarge** endpoints.
By automating capacity discovery, engineers can provision exactly the number of ml.g7e.2xlarge instances needed for peak traffic, cutting cloud spend while guaranteeing SLA‑compliant latency for generative AI services.
✅ Practical Steps
- Deploy NVIDIA Nemotron‑3 Nano 30B to a SageMaker AI endpoint using the **native vLLM container** on an **ml.g7e.2xlarge** instance.
- Run a concurrency sweep by calling **CreateAIBenchmarkJob** with your desired workload profile (input/output token counts, streaming mode).
- Analyze the benchmark output to locate the **optimal concurrency level**, **breaking point**, and compute the **right‑size factor** for fleet sizing.
Want the full story? Read the original article.
Read on AWS ML Blog ↗