Why Temperature 0 Isn't Deterministic

The article demonstrates that setting temperature to zero in large language models does not guarantee deterministic outputs; a simple formula predicts that the probability of a top‑token flip rises with sequence length, causing identical completions to diverge after roughly 100 tokens. It shows that the sampling algorithm’s internal randomness still propagates, so even with temperature = 0, the model can produce different outputs once the context exceeds a critical length. This highlights a subtle but important limitation for production pipelines that rely on reproducibility.

⚡ Key Takeaways

• The probability of a top‑token flip grows linearly with token position, reaching a 50 % chance around the 100‑token mark.
• The sampling procedure uses a deterministic RNG seed but still introduces stochasticity due to hidden state updates.
• Reproducibility is compromised beyond ~100 tokens, which can inflate inference cost if retries are needed.
• To enforce determinism, set the `do_sample=False` flag and pin the RNG seed in the inference API.
• WhyItMatters: Engineers building production LLM services must account for hidden randomness even at temperature = 0, otherwise reproducibility guarantees and debugging workflows can break unexpectedly.
• TechnicalLevel: Intermediate
• TargetAudience: ML Engineers
• PracticalSteps:
• Run `model.generate(input_ids, temperature=0.0, do_sample=False, seed=42)` to lock the RNG.
• Log the RNG seed and input prompt length to detect when token flips become likely.
• ToolsMentioned: None
• Tags: LLM, INFERENCE

Engineers building production LLM services must account for hidden randomness even at temperature = 0, otherwise reproducibility guarantees and debugging workflows can break unexpectedly.

✅ Practical Steps

1. Run `model.generate(input_ids, temperature=0.0, do_sample=False, seed=42)` to lock the RNG.
2. Log the RNG seed and input prompt length to detect when token flips become likely.

Want the full story? Read the original article.