[Paper Review] Multi-Task Deep Neural Networks for Natural Language Understanding

Xiaodong Liu, Pengcheng He+2 more|arXiv (Cornell University)|Jan 31, 2019

Topic Modeling34 references220 citations

TL;DR

MT-DNN combines multi-task learning with a pre-trained Transformer encoder to learn universal language representations, achieving state-of-the-art on GLUE and domain-adaptation gains on SNLI and SciTail.

ABSTRACT

In this paper, we present a Multi-Task Deep Neural Network (MT-DNN) for learning representations across multiple natural language understanding (NLU) tasks. MT-DNN not only leverages large amounts of cross-task data, but also benefits from a regularization effect that leads to more general representations in order to adapt to new tasks and domains. MT-DNN extends the model proposed in Liu et al. (2015) by incorporating a pre-trained bidirectional transformer language model, known as BERT (Devlin et al., 2018). MT-DNN obtains new state-of-the-art results on ten NLU tasks, including SNLI, SciTail, and eight out of nine GLUE tasks, pushing the GLUE benchmark to 82.7% (2.2% absolute improvement). We also demonstrate using the SNLI and SciTail datasets that the representations learned by MT-DNN allow domain adaptation with substantially fewer in-domain labels than the pre-trained BERT representations. The code and pre-trained models are publicly available at https://github.com/namisan/mt-dnn.

Motivation & Objective

• Leverage multi-task learning to share representations across related NLU tasks.
• Integrate a pre-trained Transformer-based encoder with multi-task objectives for universal text representations.
• Demonstrate strong performance on GLUE, SNLI, and SciTail benchmarks.
• Show improved domain adaptation with limited in-domain labeled data.

Proposed method

• Use a shared Lexicon Encoder and Transformer Encoder to produce contextualized representations across tasks.
• Attach task-specific output modules (classification, similarity, ranking) to the shared encoder.
• Train with a two-stage procedure: pretraining (masked LM and NSP) and multi-task fine-tuning across GLUE tasks.
• Adopt task-specific loss functions for classification (cross-entropy), regression (MSE), and ranking (pairwise loss).
• Incorporate a SAN-based multi-step reasoning module for pairwise text classification tasks (NLI).
• Fine-tune initialized shared layers (BERT LARGE) with multi-task objectives to obtain MT-DNN.

Experimental results

Research questions

• RQ1 How can multi-task learning improve generalization for diverse NLU tasks beyond task-specific models?
• RQ2 What is the impact of combining multi-task learning with pre-trained language models like BERT on performance across GLUE tasks?
• RQ3 Can MT-DNN enable effective domain adaptation with very limited in-domain labeled data?
• RQ4 What architectural components (e.g., SAN module, ranking formulation) contribute most to gains on NLI and QA-style tasks?

Key findings

• MT-DNN achieves new state-of-the-art results on eight of nine GLUE tasks, reaching an overall GLUE score of 82.7%.
• MT-DNN substantially improves domain adaptation: SNLI reaches 91.6% and SciTail 95.0% with full training data using MT-DNN LARGE.
• MT-DNN consistently outperforms BERT on tasks with limited in-domain data, highlighting the regularization and transfer benefits of multi-task learning.
• Without fine-tuning, MT-DNN still beats BERT LARGE on mostGLUE tasks, with CoLA being the exception due to dataset peculiarities.
• A ST-DNN variant demonstrates that task-specific output modules and loss formulations (e.g., ranking for QNLI) are crucial for gains beyond BERT baselines.
• MT-DNN enables robust domain adaptation with very small labeled datasets (0.1%–1%) showing large accuracy gains over BERT.