Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline
The AI4Bharat + NVIDIA release introduces Nemotron‑pretraining‑Indic‑v1, a 2.8 trillion‑token corpus spanning all 22 Indian Constitution languages, generated with an open, GPU‑accelerated pipeline that performs document‑level translation, quality assessment, and targeted repair while preserving equations, tables, and code. The dataset includes native‑script text for every language and romanized text for twelve, yielding 34 language‑script configurations. Engineers can directly use the corpus or adopt the published generation recipe to create additional synthetic data for their own domains. This substantially lowers the data barrier for building high‑quality Indic LLMs.
⚡ Key Takeaways
- 2.8 trillion tokens across 22 Indian languages (Nemotron‑pretraining‑Indic‑v1).
- Native‑script data for all 22 languages plus romanized text for twelve (34 language‑script configurations).
- Pipeline combines document‑level translation, quality assessment, and targeted repair, with checks to preserve equations, tables, and code.
- Open generation recipe allows teams to adapt prompts, quality checks, and source documents for custom data creation.
Providing a massive, high‑quality synthetic Indic corpus and a reusable generation pipeline enables engineers to pre‑train or continue‑train LLMs for Indian languages without the historic data scarcity bottleneck, accelerating production‑ready multilingual model development.
✅ Practical Steps
- Use the open generation recipe to adapt prompts and quality checks to your own source documents.
- Run the GPU‑accelerated pipeline to translate, assess, and repair data for your target Indic languages.
Want the full story? Read the original article.
Read on Hugging Face Blog ↗