[Paper Review] AI4Bharat-IndicNLP Corpus: Monolingual Corpora and Word Embeddings for Indic Languages

Anoop Kunchukuttan, Divyanshu Kakwani+5 more|arXiv (Cornell University)|Apr 30, 2020

Natural Language Processing Techniques23 references43 citations

TL;DR

The paper introduces the IndicNLP corpus with 2.7B words across 10 Indian languages, provides pre-trained FastText embeddings, and benchmarks (news category classification, word similarity/analogy, bilingual lexicon induction) showing improvements over public baselines.

ABSTRACT

We present the IndicNLP corpus, a large-scale, general-domain corpus containing 2.7 billion words for 10 Indian languages from two language families. We share pre-trained word embeddings trained on these corpora. We create news article category classification datasets for 9 languages to evaluate the embeddings. We show that the IndicNLP embeddings significantly outperform publicly available pre-trained embedding on multiple evaluation tasks. We hope that the availability of the corpus will accelerate Indic NLP research. The resources are available at https://github.com/ai4bharat-indicnlp/indicnlp_corpus.

Motivation & Objective

• Create large-scale monolingual corpora for 10 Indic languages reflecting contemporary use.
• Provide pre-trained word embeddings trained on the IndicNLP corpus.
• Develop downstream evaluation datasets (news category classification) and unsupervised morphology analyzers.
• Demonstrate that IndicNLP embeddings outperform publicly available embeddings on diverse NLP tasks.

Proposed method

• Collect and preprocess general-domain monolingual data from news sources and Wikipedia.
• Canonicalize Indic text, sentence-split, and tokenize using the Indic NLP Library.
• Train 300-dimensional FastText skip-gram embeddings with subword information on each language (10 epochs, window=5, min count=5, 10 negative samples).
• Evaluate embeddings on word similarity, word analogy, sentiment/text classification, and bilingual lexicon induction (BLI).
• Construct IndicNLP News Category Dataset for 9 languages and use k-NN (k=4) with mean word embeddings for classification.
• Train unsupervised Morpheme analyzers (Morfessor 2.0) and assess morphology-related improvements for SMT between Indian languages.

Experimental results

Research questions

• RQ1 Do IndicNLP embeddings outperform publicly available embeddings (FT-W, FT-WC) across intrinsic and extrinsic tasks?
• RQ2 How do monolingual IndicNLP corpora affect word similarity, analogy, sentiment, text classification, and bilingual lexicon induction performance?
• RQ3 Can IndicNLP resources support unsupervised morphology analysis and improve cross-language SMT?
• RQ4 What is the utility of the corpora for building multilingual representations and downstream NLP benchmarks?

Key findings

• IndicNLP embeddings outperform the two public baselines on multiple tasks; average word similarity (Pearson) across languages improves to 0.519 (INLP) vs 0.507 (FT-W) and 0.497 (FT-WC).
• On word analogy (Hindi subset) IndicNLP achieves 33.48% accuracy vs 19.76% (FT-W) and 32.93% (FT-WC).
• In text classification on diverse public datasets, IndicNLP embeddings show higher accuracy (average 74.73%) compared with FT-W (69.25%) and FT-WC (68.32%).
• IndicNLP News Category Dataset results show higher accuracies with INLP embeddings across languages (e.g., pa: 96.79, bn: 97.86, or: 98.07, gu: 99.02, mr: 99.37, te: 98.79, ta: 97.01, etc., averaged 97.40).
• Bilingual Lexicon Induction (BLI) with GeoMM yields higher Avg. accuracy for INLP: en→Indic 36.55 and Indic→en 44.94 (vs FT-W 25.98/33.20 and FT-WC 32.88/44.94).
• Unsupervised morphology analyzers trained on IndicNLP show improved SMT BLEU scores over word-level baselines and competitive results versus prior morph-analyzer (average BLEU: word 22.84, morph 24.21, morph (K&B, 2016) 24.57).