[Paper Review] Practical Text Classification With Large Pre-Trained Language Models

Neel Kant, Raul Puri+2 more|arXiv (Cornell University)|Dec 4, 2018

Sentiment Analysis and Opinion Mining18 references43 citations

TL;DR

The paper shows that unsupervised pretraining on a large text corpus followed by fine-tuning with Transformer and mLSTM models achieves strong practical performance on binary sentiment and multidimensional emotion classification, outperforming commercial APIs on real-world tweets and achieving competitive SemEval results.

ABSTRACT

Multi-emotion sentiment classification is a natural language processing (NLP) problem with valuable use cases on real-world data. We demonstrate that large-scale unsupervised language modeling combined with finetuning offers a practical solution to this task on difficult datasets, including those with label class imbalance and domain-specific context. By training an attention-based Transformer network (Vaswani et al. 2017) on 40GB of text (Amazon reviews) (McAuley et al. 2015) and fine-tuning on the training set, our model achieves a 0.69 F1 score on the SemEval Task 1:E-c multi-dimensional emotion classification problem (Mohammad et al. 2018), based on the Plutchik wheel of emotions (Plutchik 1979). These results are competitive with state of the art models, including strong F1 scores on difficult (emotion) categories such as Fear (0.73), Disgust (0.77) and Anger (0.78), as well as competitive results on rare categories such as Anticipation (0.42) and Surprise (0.37). Furthermore, we demonstrate our application on a real world text classification task. We create a narrowly collected text dataset of real tweets on several topics, and show that our finetuned model outperforms general purpose commercially available APIs for sentiment and multidimensional emotion classification on this dataset by a significant margin. We also perform a variety of additional studies, investigating properties of deep learning architectures, datasets and algorithms for achieving practical multidimensional sentiment classification. Overall, we find that unsupervised language modeling and finetuning is a simple framework for achieving high quality results on real-world sentiment classification.

Motivation & Objective

• Demonstrate that large-scale unsupervised language modeling combined with finetuning can solve practical text classification tasks.
• Compare Transformer and mLSTM architectures on sentiment and Plutchik-based emotion classification.
• Evaluate performance on academic datasets (SemEval) and real-world social media data against commercial APIs.
• Investigate effects of dataset size, model architecture (single head vs multihead), and active learning on performance.
• Provide ablations and analyses to understand the generalization of large pre-trained models to domain-specific contexts.

Proposed method

• Pretrain language models on 40GB of Amazon reviews using a Transformer and an mLSTM.
• Fine-tune the pretrained models on binary sentiment and Plutchik multidimensional emotion tasks with task-specific decoders.
• Use a 32,000 word-piece BPE vocabulary and a short sequence length (64) to suit tweets.
• Fine-tune the decoder f_d' with either a single binary output or a multihead setup for multi-label emotions.
• Incorporate an auxiliary language modeling loss during finetuning for the Transformer; compare with ELMo baselines.
• Apply thresholding optimized on a held-out set rather than fixed thresholds to handle class imbalance.
• Explore active learning to selectively label additional tweets and balance emotion categories.

Experimental results

Research questions

• RQ1 How effective is unsupervised pretraining on a large corpus for downstream practical sentiment and emotion classification tasks when fine-tuned on small, domain-specific labeled data?
• RQ2 Do Transformer-based models outperform mLSTM models in this transfer learning setup, especially for multidimensional emotion classification?
• RQ3 How do dataset size, decoder architecture (single head vs multihead), and active learning influence performance on imbalanced emotion categories?
• RQ4 How does the model compare to commercial sentiment/emotion APIs on real-world social media data and to SemEval benchmarks?
• RQ5 What are the challenges of applying a single model across domain contexts, and can fine-tuning with contextual data achieve robust generalization?

Key findings

• Transformer finetuning achieves a macro F1 of 0.690 on SemEval Task 1:E-c.
• On binary sentiment, the Transformer (finetune) reaches 90.9% SST accuracy and outperforms Watson and Google APIs on company tweets.
• The model achieves competitive results on emotion categories like Fear (0.73), Disgust (0.77) and Anger (0.78) in SemEval, with 0.42 (Anticipation) and 0.37 (Surprise) for rarer classes.
• Transformer generally outperforms mLSTM, particularly in multidimensional emotion classification.
• Active learning improves class balance in the labeled set (11.2% vs 8.2% positive across categories) and reduces no-emotion tweets (35.6% vs 52.1%).
• Finetuning with the Transformer (and, to a lesser extent, ELMo baselines) yields stronger results on real-world tweet datasets than off-the-shelf commercial APIs.