nemotron-halloween
MonsterHouse is an interactive Halloween projection built on NVIDIA’s Nemotron Voice Agent stack, using Nemotron 3 ASR (nemotron‑speech‑streaming‑en‑0.6b.q8_0.gguf) for speech‑to‑text, Nemotron 3.5 Bolt E4B 4GB with a LoRA adapter for dialogue generation, Magpie TTS Zeroshot for voice synthesis, and Audio2Face‑3D for real‑time mouth animation in Unreal Engine 5. The entire pipeline—listening, thinking, speaking, and animating—runs on a single workstation equipped with an RTX PRO 6000 Blackwell Max‑Q and an RTX PRO 4000 Blackwell. Engineers can replicate this end‑to‑end agent by wiring the same NVIDIA models together via the Nemotron Voice Agent coordinator.
⚡ Key Takeaways
- Nemotron 3 ASR model used: **nemotron‑speech‑streaming‑en‑0.6b.q8_0.gguf**.
- Conversational core: **Nemotron 3.5 Bolt E4B 4GB** with a LoRA adapter for personality.
- Voice synthesis: **NVIDIA Magpie TTS Zeroshot** driven by a short reference audio sample.
- Real‑time facial animation: **NVIDIA Audio2Face‑3D** integrated in **Unreal Engine 5**.
- Vision components: **NVIDIA Nemotron 3 Nano Omni** for costume description plus **YOLO** and **ByteTrack** for visitor counting and tracking.
🔧 Tools & Libraries
This demo shows that a full multimodal voice agent—speech recognition, LLM reasoning, TTS, and 3D animation—can be deployed on a single workstation, lowering the barrier for interactive AI installations in physical spaces.
✅ Practical Steps
- Install the Nemotron Voice Agent stack and load **nemotron‑speech‑streaming‑en‑0.6b.q8_0.gguf** for ASR.
- Deploy **Nemotron 3.5 Bolt E4B 4GB** with the provided LoRA adapter to generate dialogue responses.
- Connect **Magpie TTS Zeroshot** to the LLM output and pipe the audio into **Audio2Face‑3D** within Unreal Engine 5 for mouth animation.
Want the full story? Read the original article.
Read on Hugging Face Blog ↗