[Paper Review] Extractive Question Answering on Queries in Hindi and Tamil

Adhitya Thirumala, Elisa Ferracane|arXiv (Cornell University)|Sep 27, 2022

Topic Modeling4 citations

TL;DR

This paper proposes and evaluates fine-tuned transformer models—specifically XLM-RoBERTa and RoBERTa—on extractive question answering for Hindi and Tamil, using a multilingual dataset from a Kaggle competition. The best-performing model, RoBERTa fine-tuned separately on Hindi and Tamil data, achieved a 0.958 Jaccard score on Hindi and 0.829 on Tamil, significantly outperforming baseline models and demonstrating the value of language-specific fine-tuning for low-resource Indic languages.

ABSTRACT

Indic languages like Hindi and Tamil are underrepresented in the natural language processing (NLP) field compared to languages like English. Due to this underrepresentation, performance on NLP tasks (such as search algorithms) in Indic languages are inferior to their English counterparts. This difference disproportionately affects those who come from lower socioeconomic statuses because they consume the most Internet content in local languages. The goal of this project is to build an NLP model that performs better than pre-existing models for the task of extractive question-answering (QA) on a public dataset in Hindi and Tamil. Extractive QA is an NLP task where answers to questions are extracted from a corresponding body of text. To build the best solution, we used three different models. The first model is an unmodified cross-lingual version of the NLP model RoBERTa, known as XLM-RoBERTa, that is pretrained on 100 languages. The second model is based on the pretrained RoBERTa model with an extra classification head for the question answering, but we used a custom Indic tokenizer, then optimized hyperparameters and fine tuned on the Indic dataset. The third model is based on XLM-RoBERTa, but with extra finetuning and training on the Indic dataset. We hypothesize the third model will perform best because of the variety of languages the XLM-RoBERTa model has been pretrained on and the additional finetuning on the Indic dataset. This hypothesis was proven wrong because the paired RoBERTa models performed the best as the training data used was most specific to the task performed as opposed to the XLM-RoBERTa models which had much data that was not in either Hindi or Tamil.

Motivation & Objective

• Address the underrepresentation of Indic languages like Hindi and Tamil in NLP by developing effective extractive question-answering systems.
• Improve access to information for non-English speakers, particularly those from lower socioeconomic backgrounds who rely on local languages.
• Evaluate the performance of multilingual and monolingual transformer models on low-resource Indic language QA tasks.
• Investigate whether language-specific fine-tuning enhances model performance compared to multilingual pretraining on Hindi and Tamil.
• Provide a high-performing, accessible QA system for search engines to serve Hindi and Tamil speakers more effectively.

Proposed method

• Fine-tuned XLM-RoBERTa on a combined Hindi and Tamil dataset to serve as a multilingual baseline.
• Fine-tuned XLM-RoBERTa on both Hindi and Tamil data separately to improve cross-lingual generalization.
• Utilized RoBERTa models fine-tuned exclusively on Hindi and Tamil QA data, respectively, using subword tokenization via the WECHSEL package for multilingual transfer.
• Applied word-level Jaccard score as the primary evaluation metric, measuring overlap between predicted and ground-truth answers.
• Trained models for 4 epochs with a batch size of 12, a base learning rate of 2×10⁻⁵, and weight decay of 0.01.
• Used a two-step data collection process: native speakers generated questions from Wikipedia excerpts, then labeled answers were extracted from the same texts.

Experimental results

Research questions

• RQ1 How does multilingual pretraining (XLM-RoBERTa) compare to monolingual fine-tuning (RoBERTa) in extractive QA for Hindi and Tamil?
• RQ2 To what extent does language-specific fine-tuning improve performance on low-resource Indic languages compared to shared multilingual fine-tuning?
• RQ3 What are the primary error patterns in QA models for Hindi and Tamil, and how do they relate to linguistic differences such as word segmentation and grammar?
• RQ4 Can subword-embedding transfer via WECHSEL effectively adapt English-pretrained RoBERTa to Hindi and Tamil without full retraining?
• RQ5 How does the imbalance in training data (67% Hindi, 33% Tamil) affect model performance and generalization across languages?

Key findings

• The RoBERTa model fine-tuned separately on Hindi and Tamil data achieved the highest performance, with a Jaccard score of 0.958 on Hindi and 0.829 on Tamil.
• The XLM-RoBERTa baseline model achieved a Jaccard score of 0.656 on the combined Hindi and Tamil validation set.
• The XLM-RoBERTa+finetune model improved performance to a Jaccard score of 0.749 on the combined dataset, a 14.18% relative improvement over the baseline.
• The RoBERTa+Hindi/Tamil finetune model showed a 36.13% relative improvement over the XLM-RoBERTa baseline and a 19.23% improvement over XLM-RoBERTa+finetune.
• Error analysis revealed that models often included punctuation (e.g., parentheses) in answers or added grammatical elements (e.g., 'language' in Tamil), indicating limitations in syntactic generalization.
• The performance gap between models highlights the benefit of language-specific fine-tuning, especially for morphologically and orthographically distinct languages like Tamil and Hindi.