New NVIDIA Cosmos 3 Open Models for Interactive World Simulation
NVIDIA introduced four new open‑weight checkpoints in the Cosmos 3 Nano family, each with 16 B trainable parameters and licensed under OpenMDW 1.1. The models—Cosmos3‑Nano‑Transfer‑Auto, Cosmos3‑Nano‑Sim‑Auto, Cosmos3‑Nano‑Sim‑Transfer, and Cosmos3‑Nano‑Sim‑Bimanual—support multi‑view, multi‑sensor driving data generation, synchronized camera/LiDAR streaming, text‑to‑video simulation, and robot‑action video prediction respectively. These checkpoints enable researchers to build interactive world simulations for robotics and autonomous‑driving research, with the ability to fine‑tune on custom sensor rigs and domains. Engineers can now integrate open‑weight world models directly into their pipelines for offline data synthesis or causal streaming simulation.
⚡ Key Takeaways
- All four checkpoints are post‑trained variants of Cosmos 3 Nano with 16 B trainable parameters.
- Cosmos3‑Nano‑Transfer‑Auto generates synchronized video from up to 11 camera views and joint RGB‑LiDAR data using per‑camera World Scenario Map controls and optional text prompts.
- Cosmos3‑Nano‑Sim‑Auto provides causally distilled streaming generation of camera and LiDAR observations from driving controls.
- Cosmos3‑Nano‑Sim‑Transfer converts simulator controls and a text prompt into photorealistic video.
- Cosmos3‑Nano‑Sim‑Bimanual predicts video conditioned on robot or camera actions.
By releasing open‑weight, multi‑sensor world models, NVIDIA gives engineers a ready‑to‑use foundation for creating high‑fidelity simulated environments, accelerating data generation and policy evaluation for autonomous‑driving and robotics systems.
✅ Practical Steps
- Download the desired checkpoint from the provided link and run inference with Cosmos3‑Nano‑Transfer‑Auto for offline multi‑sensor data synthesis.
- Post‑train Cosmos3‑Nano‑Transfer‑Auto on your own driving dataset to adapt to your specific camera layout and LiDAR configuration before using it for offline generation or streaming simulation.
- Use the causally distilled Cosmos3‑Nano‑Sim‑Auto checkpoint to stream synchronized observations directly from driving control inputs for interactive simulation loops.
Want the full story? Read the original article.
Read on Hugging Face Blog ↗