← Back
AWS ML Blog

Deploy frontier AI models on AWS Trainium with SGLang-Neuron

•5 min read•
#inference#amazon
Level:Intermediate
For:AI Engineers
✦TL;DR

** Amazon announced SGLang‑Neuron, an out‑of‑tree plugin that lets you serve frontier LLMs such as openai/gpt‑oss‑120b on AWS Trainium 3 instances using the same SGLang server API. The plugin registers Trainium as a native PyTorch device via TorchNeuron, compiles standard PyTorch ops and Neuron Kernel Interface kernels with `torch.compile`, and supports a PD‑disaggregation workflow that moves KV‑cache from a prefill server to a decode server over EFA.

⚡ Key Takeaways

  • **Model & config:** Serves `openai/gpt-oss-120b` with `--tp-size 8`, `--ep-size 8`, `--kv-cache-dtype fp8_e4m3`, `--context-length 10240`, `--chunked-prefill-size 8192`, `--max-running-requests 64`, `--max-total-tokens 655360`, `--mem-fraction-static 0.8`.
  • **Plugin architecture:** SGLang‑Neuron is an out‑of‑tree plugin that registers the Neuron platform via SGLang’s plugin entry points and selects it with `SGLANG_PLATFORM=neuron`.
  • **Performance trade‑off:** Uses native PyTorch ops plus NKI kernels compiled with `torch.compile`; fp8 KV‑cache reduces memory bandwidth but requires Trainium hardware.
  • **Integration point:** Launch the server inside the container with `python -m sglang.launch_server …` and invoke it via the standard SGLang generation REST API (`/v1/chat/completions`).
  • **Limitation:** Requires a Trainium 3 EC2 instance and the specific `pytorch-inference-sglang-neuron` container; only models listed in the repository’s “supported models” are validated.
💡 Why It Matters

** SGLang‑Neuron gives production teams a drop‑in path to run 120 B‑parameter LLMs on Trainium with the familiar SGLang API, cutting engineering effort while leveraging fp8 KV‑cache and Tensor/Expert parallelism for cost‑effective high‑throughput inference. **TechnicalLevel:** Advanced **TargetAudience:** ML Engineers, Inference Engineers **PracticalSteps:** - Pull and start the container:…

✅ Practical Steps

  1. Pull and start the container:
  2. `docker run --rm -it --privileged --ipc=host -v /path/to/work:/work public.ecr.aws/neuron/pytorch-inference-sglang-neuron:v0.1.0-neuron-py314-sdk-2.33.0-ubuntu26.04 /bin/bash`
  3. Launch the SGLang server with Trainium:
  4. `SGLANG_PLATFORM=neuron python -m sglang.launch_server --model-path openai/gpt-oss-120b --tp-size 8 --ep-size 8 --kv-cache-dtype fp8_e4m3 --context-length 10240 --chunked-prefill-size 8192 --max-running-requests 64 --max-total-tokens 655360 --mem-fraction-static 0.8 --host 127.0.0.1 --port 30000`

Want the full story? Read the original article.

Read on AWS ML Blog ↗

More like this

Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod

AWS ML Blog•#deployment

Helping AI text detectors keep up with changing conditions

MIT News AI•#llm

Hugging Face and OpenRouter join forces to make open-weight models easy to run

Hugging Face Blog•#inference

Activation and Floating-Point Quantization

Machine Learning Mastery•#inference

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING