[Paper Review] Adversarial Multi-task Learning for Text Classification

Pengfei Liu, Xipeng Qiu+1 more|arXiv (Cornell University)|Apr 19, 2017

Topic Modeling23 references91 citations

TL;DR

The paper proposes an adversarial shared-private multi-task learning framework (ASP-MTL) to disentangle task-invariant shared features from task-specific private features in text classification, using adversarial training and orthogonality constraints, and demonstrates strong improvements across 16 datasets with transferability to new tasks.

ABSTRACT

Neural network models have shown their promising opportunities for multi-task learning, which focus on learning the shared layers to extract the common and task-invariant features. However, in most existing approaches, the extracted shared features are prone to be contaminated by task-specific features or the noise brought by other tasks. In this paper, we propose an adversarial multi-task learning framework, alleviating the shared and private latent feature spaces from interfering with each other. We conduct extensive experiments on 16 different text classification tasks, which demonstrates the benefits of our approach. Besides, we show that the shared knowledge learned by our proposed model can be regarded as off-the-shelf knowledge and easily transferred to new tasks. The datasets of all 16 tasks are publicly available at \url{http://nlp.fudan.edu.cn/data/}

Motivation & Objective

• Motivate multi-task learning for text classification and address contamination of shared features by task-specific signals.
• Propose a robust framework that enforces a disjoint, purer shared feature space via adversarial training and orthogonality constraints.
• Show improvements over baselines across 16 datasets and demonstrate transferability of shared knowledge to new tasks.

Proposed method

• Adopt LSTM-based text encoders with a fully shared, private per-task architecture and an adversarial discriminator to enforce task-invariant shared features.
• Extend binary adversarial training to multi-class, enabling joint training across many tasks and leveraging unlabeled data.
• Introduce an orthogonality loss to minimize overlap between shared and private representations: L_diff = sum_k || S^k^T H^k ||_F^2.
• Train with a combined loss L = L_Task + lambda * L_Adv + gamma * L_Diff, using a gradient reversal layer for adversarial optimization.
• Evaluate on 16 datasets from product reviews and movies, comparing against single-task LSTMs and multi-task baselines (FS-MTL, SP-MTL, ASP-MTL).

Experimental results

Research questions

• RQ1 Can adversarial training yield a purer shared representation by preventing task-specific features from leaking into the shared space?
• RQ2 Does enforcing orthogonality between shared and private spaces reduce feature redundancy and improve generalization?
• RQ3 Is the learned shared representation transferable as off-the-shelf knowledge to unseen tasks?
• RQ4 How does ASP-MTL compare to baseline multi-task approaches on a broad set of text classification tasks?
• RQ5 Can semi-supervised learning be integrated to utilize unlabeled data within this framework?

Key findings

• ASP-MTL achieves the lowest error rates across most of the 16 datasets compared with single-task baselines and other multi-task models.
• Adversarial training provides a measurable margin over SP-MTL, with an average improvement of about 4.1% over SP-MTL.
• Orthogonality constraints help reduce overlap between shared and private spaces, improving task-specific predictions and interpretability.
• Transferred shared extractors from ASP-MTL can improve performance on a new target task, especially with Bi-Channel transfer.
• The framework supports semi-supervised learning by using the task adversarial loss with unlabeled data to encourage domain-invariant sharing.