[Paper Review] Paramanu: Compact and Competitive Monolingual Language Models for Low-Resource Morphologically Rich Indian Languages

Mitodru Niyogi, Gaussier, Eric+1 more|arXiv (Cornell University)|Jan 31, 2024

Natural Language Processing Techniques4 citations

TL;DR

Paramanu introduces a family of compact, monolingual autoregressive language models for five low-resource, morphologically rich Indian languages—Bengali, Hindi, Marathi, Tamil, and Telugu—trained from scratch on single-GPU budgets under $1,000. Using morphology-aligned, low-fertility tokenizers and a novel RoPE-based sequence scaling method, the 108M–367M parameter models outperform larger multilingual models across all languages, demonstrating strong efficiency-accuracy trade-offs.

ABSTRACT

Multilingual large language models (LLMs) are expensive to pretrain and often suffer from imbalances across languages and datasets, English-centric bias, tokenizer oversegmentation for morphologically rich low-resource languages, and the curse of multilinguality. We introduce PARAMANU, the first family of Indian-only autoregressive language models trained from scratch on open-source language-specific data for the five most spoken Indian languages: Bengali, Hindi, Marathi, Tamil, and Telugu. All models are designed for affordability and are trained on a single GPU with a budget under $1,000, allowing under-resourced researchers to build competitive language models. To address low-resource challenges, we develop morphology-aligned, low-fertility tokenizers, propose an interpolation-based method for token position indices in RoPE based scaling to train longer sequences efficiently. We also create instruction-tuning datasets in Bangla that are translated to the other four languages. Despite their small size (108M-367M parameters), Paramanu achieves a strong performance-efficiency tradeoff and outperforms most larger multilingual models across all five languages. Our collection is available at https://huggingface.co/collections/mitodru/paramanu.

Motivation & Objective

• To address the limitations of multilingual LLMs in low-resource, morphologically rich Indian languages, including English-centric bias and high pretraining costs.
• To develop affordable, efficient language models that can be trained by under-resourced researchers using minimal computational resources.
• To improve performance in low-resource settings through language-specific, morphology-aligned tokenization and optimized attention mechanisms.
• To enable instruction tuning in Indian languages by creating multilingual instruction datasets translated from Bangla.
• To demonstrate that small, monolingual models can surpass larger multilingual models in low-resource language tasks.

Proposed method

• Training monolingual language models from scratch on open-source, language-specific datasets for Bengali, Hindi, Marathi, Tamil, and Telugu.
• Designing morphology-aligned, low-fertility tokenizers to reduce subword fragmentation and improve representation of morphological complexity.
• Introducing an interpolation-based method for scaling RoPE (Rotary Position Embedding) indices to enable efficient training on longer sequences without increasing context length.
• Training models on a single GPU with a total budget under $1,000 to ensure affordability and accessibility for low-resource researchers.
• Creating a multilingual instruction-tuning dataset by translating Bangla instruction data into the other four Indian languages.
• Employing a standard autoregressive transformer architecture with optimized hyperparameters for low-resource fine-tuning.

Experimental results

Research questions

• RQ1 Can compact, monolingual language models trained from scratch on low-resource Indian languages outperform larger multilingual models?
• RQ2 How does morphology-aligned tokenization improve performance in morphologically rich, low-resource languages?
• RQ3 To what extent can RoPE-based sequence scaling enable efficient training of longer sequences in small models?
• RQ4 Can instruction-tuned models in Indian languages be effectively created through translation of high-quality Bangla data?
• RQ5 What is the performance-efficiency trade-off of small, single-language models compared to larger multilingual models in low-resource settings?

Key findings

• Paramanu models, with only 108M to 367M parameters, outperform most larger multilingual models across all five Indian languages in downstream tasks.
• The morphology-aligned, low-fertility tokenizers significantly reduce subword fragmentation and improve representation of morphological complexity in Indian languages.
• The interpolation-based RoPE scaling method enables efficient training on longer sequences without increasing computational cost or context length.
• Training on a single GPU with a budget under $1,000 makes high-quality language modeling accessible to under-resourced researchers.
• The instruction-tuning dataset, translated from Bangla, enables effective zero-shot and few-shot transfer learning across Indian languages.
• Despite their small size, Paramanu models achieve competitive performance on benchmark tasks, demonstrating strong efficiency-accuracy trade-offs.