Using NVIDIA Run:ai GPU Fractioning on Amazon EKS
NVIDIA Run:ai GPU Fractioning extends MPS with Kubernetes‑integrated scheduling, per‑Pod memory contracts, enforcement, and metrics, enabling multiple Pods to share a single NVIDIA A10G Tensor Core GPU on an Amazon EKS g5.xlarge node. In the validated walkthrough each Pod is given an 8 GiB memory contract, 6 GiB allocations succeed, and a 9 GiB request is rejected, demonstrating enforced GPU‑memory limits without MIG. This allows teams to run notebooks, model evaluations, and small inference services concurrently on limited GPU resources, reducing idle wait time.
⚡ Key Takeaways
- The example runs on an Amazon EC2 **g5.xlarge** instance with an **NVIDIA A10G Tensor Core GPU**.
- GPU Fractioning Operator uses **NvFractions annotations** to specify per‑Pod memory contracts.
- A memory contract of **8 GiB** per Pod permits **6 GiB** allocations but rejects a **9 GiB** request, showing strict enforcement.
- The walkthrough leverages two of Run:ai’s four modular components: **KAI Scheduler** and **GPU Fractioning Operator**.
- The solution applies to GPUs that meet the operator’s hardware/software requirements, including those **without MIG** support.
Engineers can achieve fine‑grained GPU memory isolation on standard GPU nodes, increasing utilization and lowering cost for intermittent workloads without needing MIG‑compatible hardware.
✅ Practical Steps
- Build a custom **Amazon Linux 2023 (AL2023)** container image as described and deploy it to an EKS node group using the g5.xlarge instance type.
- Annotate Pods with **NvFractions** to request an **8 GiB** GPU‑memory contract, then verify allocation success (≤ 6 GiB) and rejection (≥ 9 GiB) via the provided per‑Pod metrics.
- Clean up the disposable environment after testing by removing the Pods and the custom node group.
Want the full story? Read the original article.
Read on AWS ML Blog ↗