[Paper Review] Automatic Spanish Translation of the SQuAD Dataset for Multilingual Question Answering

Casimiro Pio Carrino, Marta R. Costa‐jussà+1 more|arXiv (Cornell University)|Dec 11, 2019

Topic Modeling3 references42 citations

TL;DR

The paper introduces TAR, an automaticTranslate-Align-Retrieve pipeline to translate SQuAD v1.1 into Spanish (SQuAD-es), trains Spanish QA with Multilingual-BERT, and achieves state-of-the-art results on Spanish MLQA and XQuAD benchmarks.

ABSTRACT

Recently, multilingual question answering became a crucial research topic, and it is receiving increased interest in the NLP community. However, the unavailability of large-scale datasets makes it challenging to train multilingual QA systems with performance comparable to the English ones. In this work, we develop the Translate Align Retrieve (TAR) method to automatically translate the Stanford Question Answering Dataset (SQuAD) v1.1 to Spanish. We then used this dataset to train Spanish QA systems by fine-tuning a Multilingual-BERT model. Finally, we evaluated our QA models with the recently proposed MLQA and XQuAD benchmarks for cross-lingual Extractive QA. Experimental results show that our models outperform the previous Multilingual-BERT baselines achieving the new state-of-the-art value of 68.1 F1 points on the Spanish MLQA corpus and 77.6 F1 and 61.8 Exact Match points on the Spanish XQuAD corpus. The resulting, synthetically generated SQuAD-es v1.1 corpora, with almost 100% of data contained in the original English version, to the best of our knowledge, is the first large-scale QA training resource for Spanish.

Motivation & Objective

• Address the lack of large-scale Spanish QA data by generating a synthetic Spanish QA dataset from English SQuAD v1.1.
• Propose a modular TAR pipeline combining NMT, unsupervised alignment, and answer retrieval to produce Spanish QA triples.
• Evaluate Spanish QA models trained on SQuAD-es v1.1 against cross-lingual QA benchmarks MLQA and XQuAD.
• Demonstrate the effectiveness of synthetic data quality and quantity trade-offs for multilingual QA.

Proposed method

• Train an English→Spanish NMT model using a large multilingual parallel corpus and shared vocabularies.
• Develop an unsupervised context-level alignment model (eflomal/efmaral) to align source and translated contexts.
• Translate (c_src, q_src, a_src) to (c_tgt, q_tgt, a_tgt) and retrieve the corresponding answer translation using alignment.
• Handle extractive QA constraint by mapping answer spans via alignment and reordering to locate a valid a_tran in c_tran.
• Create two Spanish datasets from SQuAD v1.1: SQuAD-es v1.1 (all translated) and SQuAD-es-small (alignment-free), with cleaning heuristics.
• Fine-tune Multilingual-BERT on SQuAD-es v1.1 and evaluate on MLQA and XQuAD.

Experimental results

Research questions

• RQ1 Can automatically translated QA data (SQuAD-es) enable Spanish QA models to reach state-of-the-art performance?
• RQ2 How does the TAR-generated data compare to alignment-free synthetic data for multilingual QA training?
• RQ3 How do translation quality and alignment accuracy impact QA evaluation on cross-lingual benchmarks?

Key findings

• Models trained with TAR-generated SQuAD-es v1.1 achieve state-of-the-art F1/EM on Spanish MLQA and XQuAD benchmarks.
• On MLQA (Spanish), TAR-train + mBERT achieves 68.1 F1 and 48.3 EM, outperforming baselines and reaching close to the 2019 state-of-the-art.
• On XQuAD (Spanish), TAR-train + mBERT achieves 77.6 F1 and 61.8 EM, surpassing prior mBERT baselines and approaching the best cross-lingual models.
• Using the alignment-based retrieval increases total translated examples but yields higher error rates; removing alignment reduces size but improves precision in some settings.
• Compared to translate-train baselines, TAR-train + mBERT shows substantial improvements in F1 (11.6–14.2 points) and EM (9.8–10.9 points).
• The SQuAD-es v1.1 dataset provides a large-scale Spanish QA resource and enables robust Spanish QA performance, validating the TAR approach.