[论文解读] Practical Text Classification With Large Pre-Trained Language Models

Neel Kant, Raul Puri+2 more|arXiv (Cornell University)|Dec 4, 2018

Sentiment Analysis and Opinion Mining参考文献 18被引用 43

一句话总结

这篇论文表明，在大规模文本语料上进行自监督预训练，然后用 Transformer 和 mLSTM 模型进行微调，在二元情感和多维情感分类任务上显示出强大的实际表现，优于对实时推文的商业 API，并在 SemEval 上取得了具有竞争力的结果。

ABSTRACT

Multi-emotion sentiment classification is a natural language processing (NLP) problem with valuable use cases on real-world data. We demonstrate that large-scale unsupervised language modeling combined with finetuning offers a practical solution to this task on difficult datasets, including those with label class imbalance and domain-specific context. By training an attention-based Transformer network (Vaswani et al. 2017) on 40GB of text (Amazon reviews) (McAuley et al. 2015) and fine-tuning on the training set, our model achieves a 0.69 F1 score on the SemEval Task 1:E-c multi-dimensional emotion classification problem (Mohammad et al. 2018), based on the Plutchik wheel of emotions (Plutchik 1979). These results are competitive with state of the art models, including strong F1 scores on difficult (emotion) categories such as Fear (0.73), Disgust (0.77) and Anger (0.78), as well as competitive results on rare categories such as Anticipation (0.42) and Surprise (0.37). Furthermore, we demonstrate our application on a real world text classification task. We create a narrowly collected text dataset of real tweets on several topics, and show that our finetuned model outperforms general purpose commercially available APIs for sentiment and multidimensional emotion classification on this dataset by a significant margin. We also perform a variety of additional studies, investigating properties of deep learning architectures, datasets and algorithms for achieving practical multidimensional sentiment classification. Overall, we find that unsupervised language modeling and finetuning is a simple framework for achieving high quality results on real-world sentiment classification.

研究动机与目标

• Demonstrate that large-scale unsupervised language modeling combined with finetuning can solve practical text classification tasks.
• Compare Transformer and mLSTM architectures on sentiment and Plutchik-based emotion classification.
• Evaluate performance on academic datasets (SemEval) and real-world social media data against commercial APIs.
• Investigate effects of dataset size, model architecture (single head vs multihead), and active learning on performance.
• Provide ablations and analyses to understand the generalization of large pre-trained models to domain-specific contexts.

提出的方法

• Pretrain language models on 40GB of Amazon reviews using a Transformer and an mLSTM.
• Fine-tune the pretrained models on binary sentiment and Plutchik multidimensional emotion tasks with task-specific decoders.
• Use a 32,000 word-piece BPE vocabulary and a short sequence length (64) to suit tweets.
• Fine-tune the decoder f_d' with either a single binary output or a multihead setup for multi-label emotions.
• Incorporate an auxiliary language modeling loss during finetuning for the Transformer; compare with ELMo baselines.
• Apply thresholding optimized on a held-out set rather than fixed thresholds to handle class imbalance.
• Explore active learning to selectively label additional tweets and balance emotion categories.

实验结果

研究问题

• RQ1 How effective is unsupervised pretraining on a large corpus for downstream practical sentiment and emotion classification tasks when fine-tuned on small, domain-specific labeled data?
• RQ2 Do Transformer-based models outperform mLSTM models in this transfer learning setup, especially for multidimensional emotion classification?
• RQ3 How do dataset size, decoder architecture (single head vs multihead), and active learning influence performance on imbalanced emotion categories?
• RQ4 How does the model compare to commercial sentiment/emotion APIs on real-world social media data and to SemEval benchmarks?
• RQ5 What are the challenges of applying a single model across domain contexts, and can fine-tuning with contextual data achieve robust generalization?

主要发现

• Transformer finetuning achieves a macro F1 of 0.690 on SemEval Task 1:E-c.
• On binary sentiment, the Transformer (finetune) reaches 90.9% SST accuracy and outperforms Watson and Google APIs on company tweets.
• The model achieves competitive results on emotion categories like Fear (0.73), Disgust (0.77) and Anger (0.78) in SemEval, with 0.42 (Anticipation) and 0.37 (Surprise) for rarer classes.
• Transformer generally outperforms mLSTM, particularly in multidimensional emotion classification.
• Active learning improves class balance in the labeled set (11.2% vs 8.2% positive across categories) and reduces no-emotion tweets (35.6% vs 52.1%).
• Finetuning with the Transformer (and, to a lesser extent, ELMo baselines) yields stronger results on real-world tweet datasets than off-the-shelf commercial APIs.