[Paper Review] FQuAD2.0: French Question Answering and knowing that you know nothing

Quentin Heinrich, G. Viaud+1 more|arXiv (Cornell University)|Sep 27, 2021

Topic Modeling19 references4 citations

TL;DR

This paper introduces FQuAD2.0, the first adversarial French Question Answering dataset with 17,000+ unanswerable questions designed to be indistinguishable from answerable ones. It enables training of CamemBERT-based models to detect unanswerable questions, achieving 82.3% F1 on unanswerability detection and 83% F1 on reading comprehension, significantly advancing French NLP in open-domain QA with uncertainty awareness.

ABSTRACT

Question Answering, including Reading Comprehension, is one of the NLP research areas that has seen significant scientific breakthroughs over the past few years, thanks to the concomitant advances in Language Modeling. Most of these breakthroughs, however, are centered on the English language. In 2020, as a first strong initiative to bridge the gap to the French language, Illuin Technology introduced FQuAD1.1, a French Native Reading Comprehension dataset composed of 60,000+ questions and answers samples extracted from Wikipedia articles. Nonetheless, Question Answering models trained on this dataset have a major drawback: they are not able to predict when a given question has no answer in the paragraph of interest, therefore making unreliable predictions in various industrial use-cases. In the present work, we introduce FQuAD2.0, which extends FQuAD with 17,000+ unanswerable questions, annotated adversarially, in order to be similar to answerable ones. This new dataset, comprising a total of almost 80,000 questions, makes it possible to train French Question Answering models with the ability of distinguishing unanswerable questions from answerable ones. We benchmark several models with this dataset: our best model, a fine-tuned CamemBERT-large, achieves a F1 score of 82.3% on this classification task, and a F1 score of 83% on the Reading Comprehension task.

Motivation & Objective

• To address the critical limitation in French Question Answering models trained on FQuAD1.1, which cannot detect unanswerable questions.
• To create a high-quality, adversarially crafted French QA dataset that includes realistic unanswerable questions similar in form and content to answerable ones.
• To evaluate the impact of adversarial unanswerable questions on model performance in both unanswerability detection and reading comprehension tasks.
• To benchmark monolingual French models against multilingual models fine-tuned on English QA data, assessing cross-lingual transfer performance.
• To explore the scalability and generalization of models trained on FQuAD2.0, including the potential for model compression in industrial deployment.

Proposed method

• Annotate 17,000+ unanswerable questions adversarially to closely resemble answerable questions in linguistic form and complexity, ensuring high difficulty for models.
• Extend the original FQuAD1.1 dataset with these unanswerable samples to form FQuAD2.0, totaling nearly 80,000 questions.
• Fine-tune CamemBERT models of varying sizes (base and large) on the FQuAD2.0 training set for both answerability classification and span prediction tasks.
• Evaluate model performance on the development set using standard metrics: F1 score for reading comprehension and F1 score for unanswerable question detection (NoAns F1).
• Train and compare multilingual models (e.g., mBERT, XLM-RoBERTa) fine-tuned on English SQuAD2.0 data for zero-shot or few-shot transfer to French.
• Analyze learning curves by varying the number of adversarial samples used during training to assess data efficiency and model convergence.

Experimental results

Research questions

• RQ1 How effective are CamemBERT-based models at detecting unanswerable questions when trained on FQuAD2.0 compared to models trained only on answerable questions?
• RQ2 What is the performance drop in reading comprehension accuracy when models are trained on a mixed dataset including adversarial unanswerable questions?
• RQ3 How does the performance of monolingual French models trained on FQuAD2.0 compare to multilingual models fine-tuned on English QA data in a zero-shot setting?
• RQ4 Is FQuAD2.0 more challenging than its English counterpart SQuAD2.0 in terms of answerability detection and question ambiguity?
• RQ5 What is the impact of data scale on model performance, and how many adversarial samples are needed to achieve optimal detection capability?

Key findings

• The best-performing model, a fine-tuned CamemBERT LARGE, achieves an F1 score of 83% on the reading comprehension task and 82.3% on unanswerable question detection.
• Models trained on FQuAD2.0 show a measurable improvement in distinguishing unanswerable questions, with a significant NoAns F1 score of 82.3%, indicating strong uncertainty-aware prediction.
• The FQuAD2.0 dataset is substantially more challenging than SQuAD2.0, as evidenced by lower performance across all metrics, especially in unanswerability detection (17% lower NoAns F1 than English counterpart).
• Learning curves indicate that performance improves with more adversarial samples, suggesting that additional data could further enhance model robustness.
• Monolingual fine-tuning on FQuAD2.0 outperforms zero-shot transfer from English multilingual models, demonstrating the superiority of language-specific data for optimal performance.
• Preliminary results suggest that model compression techniques such as distillation and quantization could enable efficient deployment of large models in low-resource or high-throughput industrial settings.