[Paper Review] PMIndia -- A Collection of Parallel Corpora of Languages of India

Barry Haddow, Faheem Kirefu|arXiv (Cornell University)|Jan 27, 2020

Natural Language Processing Techniques12 references68 citations

TL;DR

PMIndia introduces a publicly available parallel corpus pairing 13 Indian languages with English, derived from PM India's news updates, with alignment methods and initial MT results.

ABSTRACT

Parallel text is required for building high-quality machine translation (MT) systems, as well as for other multilingual NLP applications. For many South Asian languages, such data is in short supply. In this paper, we described a new publicly available corpus (PMIndia) consisting of parallel sentences which pair 13 major languages of India with English. The corpus includes up to 56000 sentences for each language pair. We explain how the corpus was constructed, including an assessment of two different automatic sentence alignment methods, and present some initial NMT results on the corpus.

Motivation & Objective

• Address the data scarcity for parallel corpora in Indian languages by constructing a large English–language pair corpus.
• Leverage PM India news updates to create multilingual sentence alignments across 13 Indian languages.
• Assess alignment quality using multiple methods and provide baseline NMT results to illustrate corpus utility.
• Provide a freely accessible resource under CC-BY-4.0 to support multilingual NLP research in South Asian languages.

Proposed method

• Crawl PMIndia news updates in 13 languages and English using a custom scraper and Alcazar to extract article bodies.
• Split sentences with an extended Moses sentence splitter for all target languages.
• Align sentences using hunalign with crowd-sourced dictionaries and Vecalign when LASER embeddings are available; use the intersection when possible.
• Evaluate alignment quality via intrinsic metrics (precision, recall, F1) and a KEOPS-based human evaluation on English–Tamil subset.
• Train NMT systems on the released corpus using Marian with subword units (BPE 10000 merges) and standard low-resource MT hyperparameters.
• Report BLEU scores for English-to/from-language pairs to benchmark translation quality on the dataset.

Experimental results

Research questions

• RQ1 Can a large, openly available parallel corpus be built for India’s 13 languages paired with English from publicly accessible government content?
• RQ2 How do different automatic sentence alignment methods (hunalign, Vecalign) compare in this multilingual, low-resource setting?
• RQ3 What baseline machine translation performance (BLEU) can be achieved using this corpus for English–language pairs and vice versa?
• RQ4 What is the intrinsic alignment quality as assessed by human evaluation for selected language pairs?

Key findings

• A PMIndia corpus with up to 56,000 sentences per language pair is released, drawn from English–Indian language news updates.
• Vecalign generally agrees with hunalign at about 80% across several language pairs, with higher agreement for English–Hindi.
• Intrinsic human evaluation shows high accuracy for the intersection of both aligners, with 79% valid alignments and 94% liberal accuracy when considering non-wrong-tokenisation cases.
• NMT experiments indicate low BLEU scores for translation into Indian languages (notably Dravidian languages) and relatively better scores for English-to-Indian language directions; English–Urdu and English–Manipuri show comparatively higher results despite small data sizes.
• Dravidian and other agglutinative language features contribute to translation challenges, suggesting data quantity is not the sole factor in low performance; domain and language structure play significant roles.