[Paper Review] Learning to Ask Unanswerable Questions for Machine Reading Comprehension

Haichao Zhu, Li Dong+4 more|arXiv (Cornell University)|Jun 14, 2019

Topic Modeling46 references4 citations

TL;DR

This paper proposes a pair-to-sequence model for generating relevant unanswerable questions by editing answerable questions while conditioning on their corresponding paragraphs, enabling effective data augmentation for machine reading comprehension. The method significantly improves F1 scores by 1.9 and 1.7 points on SQuAD 2.0 using BERT-base and BERT-large models, respectively, through discriminative training signals from lexically similar, contextually grounded unanswerable questions.

ABSTRACT

Machine reading comprehension with unanswerable questions is a challenging task. In this work, we propose a data augmentation technique by automatically generating relevant unanswerable questions according to an answerable question paired with its corresponding paragraph that contains the answer. We introduce a pair-to-sequence model for unanswerable question generation, which effectively captures the interactions between the question and the paragraph. We also present a way to construct training data for our question generation models by leveraging the existing reading comprehension dataset. Experimental results show that the pair-to-sequence model performs consistently better compared with the sequence-to-sequence baseline. We further use the automatically generated unanswerable questions as a means of data augmentation on the SQuAD 2.0 dataset, yielding 1.9 absolute F1 improvement with BERT-base model and 1.7 absolute F1 improvement with BERT-large model.

Motivation & Objective

• To improve machine reading comprehension models' ability to detect unanswerable questions, which remains a key challenge despite strong performance on extractive tasks.
• To generate unanswerable questions that are lexically and contextually relevant to the paragraph, avoiding trivial or irrelevant examples.
• To develop a data augmentation strategy using automatically generated unanswerable questions to improve generalization and robustness of reading comprehension models.
• To design a training data construction method that aligns answerable and unanswerable questions via shared answer spans as pivots.
• To evaluate whether generated unanswerable questions can serve as effective negative examples to enhance model performance on SQuAD 2.0.

Proposed method

• A pair-to-sequence model is introduced to jointly encode the question and paragraph, enabling attention-based interaction and context-aware representation learning.
• The model uses a copy mechanism to incorporate words from the paragraph during generation, improving factual consistency and relevance.
• Training data is constructed by aligning answerable questions with unanswerable questions through shared answer spans, creating pivot-based pairs.
• Unanswerable questions are generated by editing answerable questions via operations like word substitution, negation, or entity replacement, conditioned on the paragraph.
• Data augmentation is applied by combining generated unanswerable questions with original SQuAD 2.0 answerable examples, followed by fine-tuning on BERT-based models.
• The approach leverages existing large-scale MRC datasets to generate a large-scale, contextually grounded unanswerable question set for training.

Experimental results

Research questions

• RQ1 Can a neural model generate unanswerable questions that are both contextually relevant and discriminative for reading comprehension models?
• RQ2 Does generating unanswerable questions through editing answerable questions with paragraph conditioning improve model generalization?
• RQ3 To what extent can automatically generated unanswerable questions enhance performance on SQuAD 2.0, especially for strong BERT-based models?
• RQ4 How does the quality of generated unanswerable questions compare to heuristic baselines like TF-IDF retrieval or rule-based methods?
• RQ5 Does increasing the volume of generated unanswerable questions further improve model performance, and is there a point of diminishing returns?

Key findings

• The pair-to-sequence model outperforms the sequence-to-sequence baseline in both automatic and human evaluations, generating more relevant and diverse unanswerable questions.
• The generated unanswerable questions yield a 1.9 absolute F1 improvement on SQuAD 2.0 when using the BERT-base model, and a 1.7 absolute F1 improvement with BERT-large.
• Data augmentation with generated unanswerable questions improves performance across multiple models, including both specialized architectures and BERT fine-tuning.
• Increasing the size of the augmentation data (up to three times) further improves performance, especially for BERT-base, indicating scalability and benefit from data diversity.
• Rule-based and TF-IDF-based baselines for unanswerable question generation provide minimal gains, highlighting the superiority of the proposed pair-to-sequence approach.
• The method effectively mitigates data imbalance and provides strong negative examples, enhancing the model’s ability to abstain when no answer is present.