← Back
AWS ML Blog

Using NVIDIA Run:ai GPU Fractioning on Amazon EKS

•16 min read•
#deployment#nvidia#compute
Level:Advanced
For:AI Engineers
✦TL;DR

NVIDIA Run:ai GPU Fractioning extends MPS with Kubernetes‑integrated scheduling, per‑Pod memory contracts, enforcement, and metrics, enabling multiple Pods to share a single NVIDIA A10G Tensor Core GPU on an Amazon EKS g5.xlarge node. In the validated walkthrough each Pod is given an 8 GiB memory contract, 6 GiB allocations succeed, and a 9 GiB request is rejected, demonstrating enforced GPU‑memory limits without MIG. This allows teams to run notebooks, model evaluations, and small inference services concurrently on limited GPU resources, reducing idle wait time.

⚡ Key Takeaways

  • The example runs on an Amazon EC2 **g5.xlarge** instance with an **NVIDIA A10G Tensor Core GPU**.
  • GPU Fractioning Operator uses **NvFractions annotations** to specify per‑Pod memory contracts.
  • A memory contract of **8 GiB** per Pod permits **6 GiB** allocations but rejects a **9 GiB** request, showing strict enforcement.
  • The walkthrough leverages two of Run:ai’s four modular components: **KAI Scheduler** and **GPU Fractioning Operator**.
  • The solution applies to GPUs that meet the operator’s hardware/software requirements, including those **without MIG** support.
💡 Why It Matters

Engineers can achieve fine‑grained GPU memory isolation on standard GPU nodes, increasing utilization and lowering cost for intermittent workloads without needing MIG‑compatible hardware.

✅ Practical Steps

  1. Build a custom **Amazon Linux 2023 (AL2023)** container image as described and deploy it to an EKS node group using the g5.xlarge instance type.
  2. Annotate Pods with **NvFractions** to request an **8 GiB** GPU‑memory contract, then verify allocation success (≤ 6 GiB) and rejection (≥ 9 GiB) via the provided per‑Pod metrics.
  3. Clean up the disposable environment after testing by removing the Pods and the custom node group.

Want the full story? Read the original article.

Read on AWS ML Blog ↗

More like this

New in AWS Quick: More capable flows, an updated web experience, and agents that work together

AWS ML Blog•#agents

Three insights you may have missed from theCUBE’s coverage of the Dell Technologies New York Symposium

SiliconANGLE AI•#deployment

Welcome Inkling 2 Mini, Cutting-edge Performance for your Local Setup

Hugging Face Blog•#llm

NVIDIA and Microsoft Expand Accelerated Computing From Azure to Windows PCs

NVIDIA Blog•#compute

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING