Deploy frontier AI models on AWS Trainium with SGLang-Neuron
** Amazon announced SGLang‑Neuron, an out‑of‑tree plugin that lets you serve frontier LLMs such as openai/gpt‑oss‑120b on AWS Trainium 3 instances using the same SGLang server API. The plugin registers Trainium as a native PyTorch device via TorchNeuron, compiles standard PyTorch ops and Neuron Kernel Interface kernels with `torch.compile`, and supports a PD‑disaggregation workflow that moves KV‑cache from a prefill server to a decode server over EFA.
⚡ Key Takeaways
- **Model & config:** Serves `openai/gpt-oss-120b` with `--tp-size 8`, `--ep-size 8`, `--kv-cache-dtype fp8_e4m3`, `--context-length 10240`, `--chunked-prefill-size 8192`, `--max-running-requests 64`, `--max-total-tokens 655360`, `--mem-fraction-static 0.8`.
- **Plugin architecture:** SGLang‑Neuron is an out‑of‑tree plugin that registers the Neuron platform via SGLang’s plugin entry points and selects it with `SGLANG_PLATFORM=neuron`.
- **Performance trade‑off:** Uses native PyTorch ops plus NKI kernels compiled with `torch.compile`; fp8 KV‑cache reduces memory bandwidth but requires Trainium hardware.
- **Integration point:** Launch the server inside the container with `python -m sglang.launch_server …` and invoke it via the standard SGLang generation REST API (`/v1/chat/completions`).
- **Limitation:** Requires a Trainium 3 EC2 instance and the specific `pytorch-inference-sglang-neuron` container; only models listed in the repository’s “supported models” are validated.
** SGLang‑Neuron gives production teams a drop‑in path to run 120 B‑parameter LLMs on Trainium with the familiar SGLang API, cutting engineering effort while leveraging fp8 KV‑cache and Tensor/Expert parallelism for cost‑effective high‑throughput inference. **TechnicalLevel:** Advanced **TargetAudience:** ML Engineers, Inference Engineers **PracticalSteps:** - Pull and start the container:…
✅ Practical Steps
- Pull and start the container:
- `docker run --rm -it --privileged --ipc=host -v /path/to/work:/work public.ecr.aws/neuron/pytorch-inference-sglang-neuron:v0.1.0-neuron-py314-sdk-2.33.0-ubuntu26.04 /bin/bash`
- Launch the SGLang server with Trainium:
- `SGLANG_PLATFORM=neuron python -m sglang.launch_server --model-path openai/gpt-oss-120b --tp-size 8 --ep-size 8 --kv-cache-dtype fp8_e4m3 --context-length 10240 --chunked-prefill-size 8192 --max-running-requests 64 --max-total-tokens 655360 --mem-fraction-static 0.8 --host 127.0.0.1 --port 30000`
Want the full story? Read the original article.
Read on AWS ML Blog ↗