← Back
AWS ML Blog

Build a model improvement flywheel with Amazon SageMaker HyperPod Data Capture

•30 min read•
#deployment#llm
Level:Advanced
For:AI Engineers
✦TL;DR

Amazon SageMaker HyperPod now includes a built‑in Data Capture feature that records live inference inputs and outputs to Amazon S3 across three independent tiers with minimal latency and no impact on model availability. The post demonstrates a full flywheel on a single HyperPod cluster: capture production traffic, query it with Amazon Athena, label records via a SQL policy, and fine‑tune the model on the newly captured data, showing measurable accuracy improvement on previously unseen inputs. Engineers can enable Tier 3 capture to store full payloads for training while using AWS KMS encryption and optional PII redaction to meet compliance.

⚡ Key Takeaways

  • Data Capture offers three tiers (Endpoint, Load balancer, Model pod) and stores full request/response payloads at Tier 1 and Tier 3.
  • Tier 3 uses the HyperPod inference operator stack (NGINX + OTel collector + S3 uploader sidecars) to write payloads directly to S3.
  • Enabling both Tier 1 and Tier 3 stores each record twice, so choose tiers based on compliance vs. training needs.
  • Configuration fields include sampling %, input/output selection, AWS KMS key ID, buffer size, max payload size, and content types.
  • Requires a SageMaker HyperPod cluster on Amazon EKS with the inference operator and the training‑operator add‑on installed, plus two NVIDIA GPUs (ml.g6e.8xlarge for serving, ml.g6e.xlarge for training, each with a 48 GB L40S).
💡 Why It Matters

By automating the capture‑label‑fine‑tune loop inside HyperPod, teams can continuously adapt foundation models to domain‑specific drift without manual data pipelines, accelerating time‑to‑value for production AI services.

✅ Practical Steps

  1. Deploy a SageMaker HyperPod cluster on Amazon EKS with the inference operator and install the amazon‑sagemaker‑hyperpod‑training‑operator add‑on.
  2. Enable Tier 3 Data Capture in the endpoint’s dataCapture block, specifying sampling % and AWS KMS encryption as needed.
  3. After capture, query the S3‑stored records with Amazon Athena, apply a SQL policy to label new‑product cases, and launch a fine‑tuning job using the training operator on the second GPU.

Want the full story? Read the original article.

Read on AWS ML Blog ↗

More like this

Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod

AWS ML Blog•#deployment

Introducing Unity Gateway Insights

Databricks Blog•#deployment

Introducing Gemini 4 Neon

Google DeepMind•#llm

DaisyRose-27B: Pushing the Frontier of Post-Training Efficiency

Hugging Face Blog•#llm

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING