Build a model improvement flywheel with Amazon SageMaker HyperPod Data Capture
Amazon SageMaker HyperPod now includes a built‑in Data Capture feature that records live inference inputs and outputs to Amazon S3 across three independent tiers with minimal latency and no impact on model availability. The post demonstrates a full flywheel on a single HyperPod cluster: capture production traffic, query it with Amazon Athena, label records via a SQL policy, and fine‑tune the model on the newly captured data, showing measurable accuracy improvement on previously unseen inputs. Engineers can enable Tier 3 capture to store full payloads for training while using AWS KMS encryption and optional PII redaction to meet compliance.
⚡ Key Takeaways
- Data Capture offers three tiers (Endpoint, Load balancer, Model pod) and stores full request/response payloads at Tier 1 and Tier 3.
- Tier 3 uses the HyperPod inference operator stack (NGINX + OTel collector + S3 uploader sidecars) to write payloads directly to S3.
- Enabling both Tier 1 and Tier 3 stores each record twice, so choose tiers based on compliance vs. training needs.
- Configuration fields include sampling %, input/output selection, AWS KMS key ID, buffer size, max payload size, and content types.
- Requires a SageMaker HyperPod cluster on Amazon EKS with the inference operator and the training‑operator add‑on installed, plus two NVIDIA GPUs (ml.g6e.8xlarge for serving, ml.g6e.xlarge for training, each with a 48 GB L40S).
By automating the capture‑label‑fine‑tune loop inside HyperPod, teams can continuously adapt foundation models to domain‑specific drift without manual data pipelines, accelerating time‑to‑value for production AI services.
✅ Practical Steps
- Deploy a SageMaker HyperPod cluster on Amazon EKS with the inference operator and install the amazon‑sagemaker‑hyperpod‑training‑operator add‑on.
- Enable Tier 3 Data Capture in the endpoint’s dataCapture block, specifying sampling % and AWS KMS encryption as needed.
- After capture, query the S3‑stored records with Amazon Athena, apply a SQL policy to label new‑product cases, and launch a fine‑tuning job using the training operator on the second GPU.
Want the full story? Read the original article.
Read on AWS ML Blog ↗