[Paper Review] Learning Robust, Transferable Sentence Representations for Text Classification

Wasi Uddin Ahmad, Xueying Bai+2 more|arXiv (Cornell University)|Sep 28, 2018

Topic Modeling38 references3 citations

TL;DR

This paper proposes a multi-task learning framework that jointly trains sentence encoders on multiple text classification datasets to learn robust, transferable sentence representations by separating generic linguistic features from task-specific knowledge. By combining these representations with pre-trained contextualized word embeddings (ELMo) and another multi-task sentence encoder (GenSen), the method achieves state-of-the-art performance on 15 transfer learning tasks and 10 linguistic probing tasks, demonstrating superior sample efficiency and generalization.

ABSTRACT

Despite deep recurrent neural networks (RNNs) demonstrate strong performance in text classification, training RNN models are often expensive and requires an extensive collection of annotated data which may not be available. To overcome the data limitation issue, existing approaches leverage either pre-trained word embedding or sentence representation to lift the burden of training RNNs from scratch. In this paper, we show that jointly learning sentence representations from multiple text classification tasks and combining them with pre-trained word-level and sentence level encoders result in robust sentence representations that are useful for transfer learning. Extensive experiments and analyses using a wide range of transfer and linguistic tasks endorse the effectiveness of our approach.

Motivation & Objective

• Address the challenge of limited annotated data in text classification by learning transferable sentence representations without training from scratch.
• Improve the robustness and transferability of sentence representations by jointly learning from multiple heterogeneous text classification datasets.
• Separate generic linguistic features (e.g., syntax, semantics) from task-specific knowledge using adversarial training in a multi-task learning framework.
• Enhance performance by fusing sentence-level representations with pre-trained contextualized word embeddings (ELMo) and a complementary multi-task sentence encoder (GenSen).

Proposed method

• Train a multi-task sentence encoder on three large-scale text classification datasets: SNLI (natural language inference), QQP (question paraphrase), and a third diverse text classification dataset to ensure domain diversity.
• Employ an adversarial training strategy to disentangle shared (generic) representations from task-specific representations, improving generalization and transferability.
• Use a shared encoder for generic linguistic features and private encoders for domain-specific knowledge, with attention-based fusion to dynamically weight contributions per transfer task.
• Combine the MTL-based sentence encoder with ELMo (contextualized word embeddings) and GenSen (a multi-task sentence encoder) to form a unified sentence representation model.
• Apply the unified encoder to downstream transfer tasks using a simple classifier head, enabling end-to-end fine-tuning with minimal labeled data.
• Use linguistic probing tasks to analyze which linguistic properties (e.g., syntax, semantics) are captured by the learned representations.

Experimental results

Research questions

• RQ1 Can jointly learning sentence representations across multiple text classification tasks improve robustness and transferability compared to single-task pretraining?
• RQ2 How effective is adversarial training in disentangling generic linguistic features from task-specific knowledge in sentence encoders?
• RQ3 To what extent do combined sentence and contextualized word representations outperform individual components on transfer learning and probing tasks?
• RQ4 How does the proposed method compare in sample efficiency when labeled data is scarce?
• RQ5 Which linguistic properties (e.g., syntactic, semantic) are best captured by the proposed unified sentence encoder?

Key findings

• The MTL-based sentence encoder (Sent2Vec) outperforms single-task models like InferSent and achieves state-of-the-art results on 15 transfer learning tasks, with a mean accuracy of 91.0 on the SST-2 dataset when combined with ELMo and GenSen.
• The unified encoder (Sent2Vec + GenSen + ELMo) achieves the highest performance across all 10 linguistic probing tasks, with a mean accuracy of 73.3 on CoordInv and 71.1 on SOMO, outperforming ELMo and InferSent.
• On the SST-2 and SICK-E tasks, the proposed method shows significantly higher sample efficiency, achieving better performance with less than 5,000 labeled examples compared to training from scratch.
• Adversarial training effectively separates generic and task-specific representations, as shown by lower attention weights on the shared encoder in most transfer tasks, indicating better disentanglement.
• The SNLI-specific encoder receives higher attention weights across most transfer tasks, confirming its strong generalization capacity for diverse text classification tasks.
• The combination of MTL sentence encoders with ELMo and GenSen yields the best overall performance, demonstrating that complementary learning signals from different pretraining objectives enhance robustness and transferability.