Multi-Turn Reinforcement Learning on Amazon SageMaker AI: A hands-on guide

Multi‑Turn Reinforcement Learning (MTRL) on Amazon SageMaker AI lets you fine‑tune a policy model using only episode‑level rewards from an agent hosted on Amazon Bedrock AgentCore. In the guide, the 20‑b parameter open‑source model **gpt‑oss‑20b** rose from **61 % to 84 %** solve rate on held‑out Wordle words at default reasoning effort, and from **15 % to 69 %** at a low‑effort setting, for roughly **$339** in training tokens. The end‑to‑end pipeline deploys the agent and reward function to AgentCore, streams rollouts via a Lambda forwarder, trains with SageMaker AI using Group Relative Policy Optimization (GRPO), and monitors with MLflow. Engineers can apply the same flow to any multi‑step business workflow where only final success signals are available.

⚡ Key Takeaways

• gpt‑oss‑20b solve rate improved from 61 % → 84 % (default) and 15 % → 69 % (low effort) for $339 in tokens.
• Uses **Group Relative Policy Optimization (GRPO)** as the RL algorithm across rollouts.
• Trade‑off: high‑effort reasoning yields higher baseline performance, but low‑effort training achieves a larger relative gain (15 %→69 %).
• Integration point: policy model is called through SageMaker Job Runtime’s **OpenAI‑compatible endpoint** from the AgentCore‑hosted Strands agent.
• Prerequisite: the agent, environment, and reward function must be deployed to **Amazon Bedrock AgentCore** (or alternative infra via Lambda forwarder).

By leveraging episode‑level rewards, teams can cheaply adapt small LLMs to complex, multi‑step internal processes without needing dense supervision, cutting both annotation cost and inference latency compared to using large public models.

✅ Practical Steps

1. Deploy your agent, Wordle‑style environment, and reward function to **Amazon Bedrock AgentCore** (or via Lambda to EKS/EC2/Fargate).
2. Submit a SageMaker AI training job that points to the AgentCore runtime, using the **OpenAI‑compatible endpoint** for the policy model and selecting **GRPO** as the optimizer.
3. Monitor rollout rewards and training metrics in **MLflow** as described, then evaluate the trained model at serving time.

Want the full story? Read the original article.