[Paper Review] SentiLR: Linguistic Knowledge Enhanced Language Representation for Sentiment Analysis.

Pei Ke, Haozhe Ji+3 more|arXiv (Cornell University)|Nov 6, 2019

Topic ModelingComputer Science51 references27 citations

TL;DR

SentiLR is a novel language representation model that enhances sentiment analysis by integrating word-level linguistic knowledge—part-of-speech tagging and SentiWordNet-based sentiment polarity—into pre-training. It introduces a label-aware masked language model (LA-MLM) with two subtasks: recovering linguistic knowledge given sentence-level sentiment labels and predicting labels using knowledge-enhanced context, achieving state-of-the-art performance on sentence- and aspect-level sentiment analysis tasks.

ABSTRACT

Most of the existing pre-trained language representation models neglect to consider the linguistic knowledge of texts, whereas we argue that such knowledge can promote language understanding in various NLP tasks. In this paper, we propose a novel language representation model called SentiLR, which introduces word-level linguistic knowledge including part-of-speech tag and prior sentiment polarity from SentiWordNet to benefit the downstream tasks in sentiment analysis. During pre-training, we first acquire the prior sentiment polarity of each word by querying the SentiWordNet dictionary with its part-of-speech tag. Then, we devise a new pre-training task called label-aware masked language model (LA-MLM) consisting of two subtasks: 1) word knowledge recovering given the sentence-level label; 2) sentence-level label prediction with linguistic knowledge enhanced context. Experiments show that SentiLR achieves state-of-the-art performance on several sentence-level / aspect-level sentiment analysis tasks by fine-tuning, and also obtain comparative results on general language understanding tasks.

Motivation & Objective

• To address the limitation of existing pre-trained language models in leveraging linguistic knowledge for improved sentiment understanding.
• To investigate whether integrating part-of-speech tags and prior sentiment polarity from SentiWordNet enhances representation learning for sentiment analysis.
• To develop a new pre-training objective that jointly utilizes linguistic knowledge and sentence-level sentiment labels to improve downstream performance.
• To evaluate the effectiveness of the proposed model on both sentiment-specific and general language understanding tasks.

Proposed method

• Obtaining prior sentiment polarity for each word by querying SentiWordNet using its part-of-speech tag.
• Designing a label-aware masked language model (LA-MLM) with two subtasks: (1) recovering masked linguistic knowledge (POS and sentiment polarity) given the sentence-level label, and (2) predicting the sentence-level label using context enriched with linguistic knowledge.
• Integrating linguistic knowledge into the input representation by conditioning the model on POS tags and sentiment scores during pre-training.
• Pre-training SentiLR on large-scale text using the LA-MLM objective to align contextual representations with both linguistic features and sentiment labels.
• Fine-tuning the resulting model on downstream sentiment analysis tasks, including sentence-level and aspect-level sentiment classification.
• Evaluating performance on sentiment analysis benchmarks and general NLP tasks to assess transferability and effectiveness.

Experimental results

Research questions

• RQ1 Can integrating part-of-speech tags and SentiWordNet sentiment polarity improve language representation for sentiment analysis?
• RQ2 Does a label-aware masked language modeling objective that leverages linguistic knowledge lead to better performance on sentiment classification tasks?
• RQ3 How does SentiLR compare to existing pre-trained models on both sentiment-specific and general language understanding benchmarks?
• RQ4 To what extent does the incorporation of linguistic knowledge enhance zero-shot or few-shot transfer learning in sentiment analysis?

Key findings

• SentiLR achieves state-of-the-art performance on multiple sentence-level sentiment analysis benchmarks after fine-tuning.
• The model also obtains competitive results on aspect-level sentiment analysis tasks, demonstrating its effectiveness in fine-grained sentiment understanding.
• SentiLR performs comparably to strong general-purpose language models on general language understanding tasks, indicating broad transferability.
• The ablation study confirms that both the linguistic knowledge integration and the LA-MLM pre-training objective contribute significantly to performance gains.
• The use of SentiWordNet-derived sentiment polarity and POS tags enhances the model's ability to capture sentiment-relevant features in context.
• The label-aware pre-training objective improves the alignment between contextual representations and sentiment labels, leading to better downstream accuracy.