← Back
Google Research

The sleep framework: A new way for LLMs to retain knowledge in continual learning

•5 min read•
#llm#agents
The sleep framework: A new way for LLMs to retain knowledge in continual learning
Level:Advanced
For:AI Engineers
✦TL;DR

The paper introduces a “sleep” phase for continual learning in large language models, allowing them to periodically consolidate short‑term context into stable long‑term parameters. By alternating active interaction with offline memory consolidation, the approach enables LLMs to acquire new facts, retain up to 10 M tokens of context, and improve complex reasoning without changing the core architecture. The method leverages dynamic parameter expansion via Mixture‑of‑Experts (MoE) blocks and a two‑stage consolidation/dreaming process to mitigate catastrophic forgetting. This provides a practical pathway for deploying LLMs that can continuously learn post‑deployment.

⚡ Key Takeaways

  • Retains long‑context knowledge up to **10 M tokens** after sleep consolidation.
  • Uses **Mixture‑of‑Experts (MoE) memory blocks** and **parameter expansion** to store new knowledge.
  • Balances **catastrophic forgetting vs. continual adaptation** by isolating new low‑rank experts in lower‑frequency memory.
  • Integration requires no changes to the model’s core codebase; the sleep phase runs as an **offline post‑training cycle**.
  • Requires **pre‑allocated MoE memory** and a schedule to trigger periodic sleep phases.
💡 Why It Matters

Engineers can keep deployed LLMs up‑to‑date and improve reasoning without costly retraining pipelines, extending the useful lifespan of models in production.

✅ Practical Steps

  1. Incorporate a periodic offline “sleep” routine that unlocks pre‑allocated MoE memory blocks and runs self‑generated practice examples to update weights.

Want the full story? Read the original article.

Read on Google Research ↗

More like this

Censored vs. Truncated Regression in Python, Explained

Towards Data Science•#llm

New in AWS Quick: More capable flows, an updated web experience, and agents that work together

AWS ML Blog•#agents

How Google Duplex evolved to also call businesses for you in Gemini

Google Research•#agents

Q&A: Enabling more human-like interactions and understanding in multimodal foundation models

MIT News AI•#llm

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING