[Paper Review] Recurrent Neural Network for Text Classification with Multi-Task Learning

Pengfei Liu, Xipeng Qiu+1 more|arXiv (Cornell University)|May 17, 2016

Topic Modeling19 references965 citations

TL;DR

The paper introduces three RNN-based multi-task architectures that share information across related text classification tasks, improving performance over single-task learning.

ABSTRACT

Neural network based methods have obtained great progress on a variety of natural language processing tasks. However, in most previous works, the models are learned based on single-task supervised objectives, which often suffer from insufficient training data. In this paper, we use the multi-task learning framework to jointly learn across multiple related tasks. Based on recurrent neural network, we propose three different mechanisms of sharing information to model text with task-specific and shared layers. The entire network is trained jointly on all these tasks. Experiments on four benchmark text classification tasks show that our proposed models can improve the performance of a task with the help of other related tasks.

Motivation & Objective

• Motivate learning text representations with limited labeled data by leveraging related tasks through multi-task learning.
• Propose three sharing architectures to enable varying levels of information exchange between tasks.
• Demonstrate that joint training across tasks improves performance over single-task baselines.
• Explore the trade-offs between shared representations and task-specific components.
• Provide insights into gating mechanisms that control information flow between shared and task-specific layers.

Proposed method

• Build on recurrent neural networks (RNNs) with LSTM units to model variable-length text inputs.
• Propose three architectures: Model-I Uniform-Layer, Model-II Coupled-Layer, and Model-III Shared-Layer for multi-task sharing.
• Introduce gating mechanisms to selectively control information flow across shared and task-specific layers.
• Train the joint network with a combined loss: sum of task losses weighted by lambda parameters (phi = sum_m lambda_m L_hat^ (m)).
• Initialize and train embeddings (shared and task-specific) with word2vec pretraining and fine-tune during supervised learning.
• Evaluate using four text classification datasets (SST-1, SST-2, SUBJ, IMDB) and compare against single-task LSTM and other neural baselines.

Experimental results

Research questions

• RQ1 Does multi-task learning with shared and/or coupled task architectures improve text classification performance compared to single-task LSTM baselines?
• RQ2 Which sharing mechanism (uniform, coupled, or shared-layer with gating) yields the best performance across diverse datasets?
• RQ3 How does unsupervised pre-training of the shared layer and fine-tuning affect multi-task performance?
• RQ4 What is the impact of multi-task learning on different data scales (sentence vs. document level)?
• RQ5 How do the proposed models compare to state-of-the-art neural models on standard benchmarks?

Key findings

• All three multi-task models outperform the single-task LSTM baseline.
• Uniform-layer architecture yields average improvement around 0.8% (0.8% before fine-tuning; 2.0% after fine-tuning).
• Coupled-layer architecture shows improvements across task pairs, with average gains around 1.4% (and up to 2.3% for SST1-SST2 pair).
• Shared-layer architecture achieves average gains around 1.4%, with additional gains from language-model pre-training (+0.5%) and fine-tuning (+0.9%).
• The multi-task model with shared layer (Model-III) achieves competitive results against several state-of-the-art neural models, with Multi-Task achieving 49.6% on SST-1, 87.9% on SST-2, 94.1% on SUBJ, and 91.3% on IMDB (table comparisons).
• The most complex Model-III costs about 2.5x the training time of vanilla LSTM but converges faster.