[Paper Review] Multi-Task Learning for Machine Reading Comprehension.

Yichong Xu, Xiaodong Liu+3 more|arXiv (Cornell University)|Sep 18, 2018

Topic Modeling14 references11 citations

TL;DR

This paper proposes a multi-task learning framework that jointly trains a machine reading comprehension (MRC) model across multiple datasets from diverse domains, leveraging out-domain data to learn robust and general contextual representations. The approach achieves significant performance gains over state-of-the-art models on SQuAD, MS MARCO, NewsQA, and other benchmarks, demonstrating orthogonal improvement to existing pre-trained representations like word embeddings and language models.

ABSTRACT

We propose a multi-task learning framework to jointly train a Machine Reading Comprehension (MRC) model on multiple datasets across different domains. Key to the proposed method is to learn robust and general contextual representations with the help of out-domain data in a multi-task framework. Empirical study shows that the proposed approach is orthogonal to the existing pre-trained representation models, such as word embedding and language models. Experiments on the Stanford Question Answering Dataset (SQuAD), the Microsoft MAchine Reading COmprehension Dataset (MS MARCO), NewsQA and other datasets show that our multi-task learning approach achieves significant improvement over state-of-the-art models in most MRC tasks.

Motivation & Objective

• To improve the generalization and robustness of machine reading comprehension models across diverse domains.
• To investigate whether out-domain training data can enhance contextual representation learning in MRC.
• To develop a multi-task learning framework that jointly optimizes performance across multiple MRC datasets.
• To evaluate the compatibility and complementary benefits of multi-task learning with existing pre-trained representation models.
• To achieve state-of-the-art performance on multiple MRC benchmarks through joint training.

Proposed method

• The framework jointly trains a single MRC model on multiple datasets from different domains, including SQuAD, MS MARCO, and NewsQA.
• It leverages out-domain data during training to learn more robust and general contextual representations.
• The method is designed to be orthogonal to existing pre-trained representation models such as word embeddings and language models.
• The model is trained end-to-end with shared and task-specific components to balance domain-specific and generalizable features.
• The multi-task objective function combines loss signals from all datasets to jointly optimize for comprehension across domains.
• The architecture enables transfer of knowledge from high-resource domains to low-resource or out-of-domain settings.

Experimental results

Research questions

• RQ1 Can joint training across multiple MRC datasets from different domains improve model generalization?
• RQ2 Does incorporating out-domain data enhance the quality of contextual representations in MRC models?
• RQ3 How does multi-task learning compare to fine-tuning on individual datasets in terms of performance and robustness?
• RQ4 To what extent is the proposed method compatible with and complementary to existing pre-trained representation models?
• RQ5 Can the multi-task framework achieve state-of-the-art results across diverse MRC benchmarks?

Key findings

• The proposed multi-task learning approach achieves significant performance improvements over state-of-the-art models on SQuAD, MS MARCO, and NewsQA.
• The method demonstrates orthogonal gains when combined with existing pre-trained representation models like word embeddings and language models.
• Out-domain data contributes meaningfully to learning robust and general contextual representations in the MRC model.
• The framework improves model generalization across diverse domains without requiring domain-specific adaptation.
• Empirical results show consistent gains across multiple datasets, indicating the effectiveness of joint training.
• The approach is effective even when training on low-resource or out-of-domain datasets, suggesting strong transfer capabilities.