[Paper Review] Bridging the Gap between Language Model and Reading Comprehension: Unsupervised MRC via Self-Supervision

Ning Bian, Xianpei Han+4 more|arXiv (Cornell University)|Jul 19, 2021

Topic Modeling34 references4 citations

TL;DR

This paper proposes a self-supervised framework for unsupervised machine reading comprehension (MRC) that bridges the gap between pre-trained language models (PLMs) and MRC by introducing a novel pretext task, Spotting-MLM, which trains models to identify and copy answer spans from documents without labeled data. The method improves robustness via question rewriting in inference and achieves state-of-the-art performance on standard MRC benchmarks, outperforming data synthetization baselines by up to 34.7 F1 points on adversarial datasets.

ABSTRACT

Despite recent success in machine reading comprehension (MRC), learning high-quality MRC models still requires large-scale labeled training data, even using strong pre-trained language models (PLMs). The pre-training tasks for PLMs are not question-answering or MRC-based tasks, making existing PLMs unable to be directly used for unsupervised MRC. Specifically, MRC aims to spot an accurate answer span from the given document, but PLMs focus on token filling in sentences. In this paper, we propose a new framework for unsupervised MRC. Firstly, we propose to learn to spot answer spans in documents via self-supervised learning, by designing a self-supervision pretext task for MRC - Spotting-MLM. Solving this task requires capturing deep interactions between sentences in documents. Secondly, we apply a simple sentence rewriting strategy in the inference stage to alleviate the expression mismatch between questions and documents. Experiments show that our method achieves a new state-of-the-art performance for unsupervised MRC.

Motivation & Objective

• To address the data-hungry nature of MRC models, which rely on large-scale human-annotated datasets like SQuAD.
• To close the gap between pre-trained language models (PLMs), trained on token-filling tasks, and MRC, which requires context-sensitive span spotting.
• To develop a self-supervised method that learns span spotting ability without any manually-labeled or synthetic training data.
• To improve robustness in unsupervised MRC by resolving expression mismatch between questions and documents during inference.
• To demonstrate that self-supervised learning can be complementary to data synthetization, enhancing performance even with limited labeled data.

Proposed method

• Introduces Spotting-MLM, a self-supervision pretext task where a repeating informative span in a document is masked and the model must copy a correct span from the same passage to fill it.
• The task requires modeling deep interactions between sentences, as the model must locate and retrieve a semantically relevant span rather than predict a single token.
• During inference, questions are rewritten into declarative sentences to align with the training distribution of unannotated text, reducing expression mismatch.
• The framework uses a standard transformer-based model (e.g., BERT) fine-tuned on the self-supervised Spotting-MLM task using only unannotated text corpora.
• The method is combined with data synthetization techniques by fine-tuning the self-supervised model on synthetic MRC data, improving performance further.
• The approach avoids template-based or data-driven question generation, instead relying on self-supervision to learn span spotting directly from raw text.

Experimental results

Research questions

• RQ1 Can a self-supervised pretext task effectively learn context-sensitive span spotting for unsupervised MRC without any labeled data?
• RQ2 Does rewriting questions into declarative forms during inference reduce expression mismatch and improve robustness in unsupervised MRC?
• RQ3 How does the proposed self-supervised method compare to data synthetization baselines in terms of performance and robustness on standard and adversarial MRC benchmarks?
• RQ4 Can the self-supervised model be effectively combined with synthetic data to further improve performance in low-resource settings?
• RQ5 To what extent can self-supervised pre-training reduce the dependency on large-scale human-annotated MRC datasets?

Key findings

• The proposed self-supervised method achieves a new state-of-the-art performance on unsupervised MRC, outperforming data synthetization baselines by up to 34.7 F1 points on adversarial datasets.
• On the SQuAD and NewsQA datasets, the method achieves EM/F1 scores of 78.8/86.8 and 71.0/78.8 respectively, significantly outperforming existing unsupervised baselines.
• On adversarial datasets, the method improves F1 by 26.0 and 34.7 points over the best-performing baselines, demonstrating superior robustness.
• In low-resource settings, fine-tuning the self-supervised model on just 1% of the labeled training data achieves an average F1 of 71.0, outperforming BERT fine-tuned on the same data.
• Combining the self-supervised model with data synthetization leads to further performance gains, showing complementarity between the two approaches.
• The sentence rewriting strategy in inference significantly improves generalization, reducing the impact of expression mismatch between questions and documents.