[Paper Review] SentiLARE: Sentiment-Aware Language Representation Learning with Linguistic Knowledge

Pei Ke, Haozhe Ji+3 more|arXiv (Cornell University)|Nov 6, 2019

Topic Modeling46 references7 citations

TL;DR

SentiLARE is a sentiment-aware pre-trained language model that integrates word-level linguistic knowledge—part-of-speech tags and context-aware sentiment polarity from SentiWordNet—into BERT-style architectures via a novel pre-training task called label-aware masked language modeling. It achieves state-of-the-art performance on multiple sentiment analysis benchmarks by fusing local linguistic knowledge with global sentence representations through early fusion and late supervision.

ABSTRACT

Most of the existing pre-trained language representation models neglect to consider the linguistic knowledge of texts, which can promote language understanding in NLP tasks. To benefit the downstream tasks in sentiment analysis, we propose a novel language representation model called SentiLARE, which introduces word-level linguistic knowledge including part-of-speech tag and sentiment polarity (inferred from SentiWordNet) into pre-trained models. We first propose a context-aware sentiment attention mechanism to acquire the sentiment polarity of each word with its part-of-speech tag by querying SentiWordNet. Then, we devise a new pre-training task called label-aware masked language model to construct knowledge-aware language representation. Experiments show that SentiLARE obtains new state-of-the-art performance on a variety of sentiment analysis tasks.

Motivation & Objective

• To address the limitation of existing pre-trained models in ignoring word-level linguistic knowledge such as part-of-speech tags and sentiment polarity.
• To improve sentiment understanding in downstream NLP tasks by integrating contextual linguistic knowledge into pre-trained representations.
• To design a pre-training task that effectively connects word-level linguistic knowledge with sentence-level sentiment labels.
• To evaluate whether the proposed method generalizes beyond sentiment analysis to other NLP tasks involving sentiment.

Proposed method

• A context-aware sentiment attention mechanism is used to infer the sentiment polarity of each word by querying SentiWordNet using the word and its part-of-speech tag.
• The model introduces a novel pre-training task called label-aware masked language modeling, which includes two sub-tasks: early fusion (integrating sentiment labels into input embeddings) and late supervision (using labels in the output layer).
• Word-level linguistic knowledge (part-of-speech and sentiment polarity) is injected into the input embeddings to enrich contextual representations.
• The model is fine-tuned on downstream sentiment analysis tasks using the enriched representations from the pre-training phase.
• The label-aware masked language modeling task jointly optimizes for masked word prediction, part-of-speech tagging, sentiment polarity prediction, and sentence-level sentiment label prediction.
• The method is evaluated on RoBERTa, BERT, and ALBERT to assess transferability across different pre-trained model architectures.

Experimental results

Research questions

• RQ1 Can integrating context-aware linguistic knowledge (POS and sentiment polarity) into pre-trained models improve performance on sentiment analysis tasks?
• RQ2 How effective is the proposed label-aware masked language modeling pre-training task in connecting word-level knowledge with sentence-level sentiment?
• RQ3 Does the integration of linguistic knowledge generalize beyond sentiment analysis to other NLP tasks involving sentiment?
• RQ4 Can the proposed method outperform existing state-of-the-art models on both sentence-level and aspect-level sentiment analysis benchmarks?

Key findings

• SentiLARE achieves new state-of-the-art performance on multiple sentiment analysis benchmarks, including SST-2, SST-Less, SST-More, and several aspect-level tasks.
• On the SST-More subset, SentiLARE outperforms RoBERTa by 4.11% in accuracy and SentiLARE-EF-LS by 2.93%, demonstrating the effectiveness of its pre-training task.
• SentiLARE based on BERT and ALBERT outperforms their vanilla counterparts across all evaluated tasks, confirming the method's transferability to different pre-trained models.
• The model generalizes to non-sentiment tasks: on text matching tasks (story ending prediction, textual entailment, semantic textual similarity), SentiLARE improves RoBERTa’s accuracy by 1.52% and correlation by up to 0.89 points.
• The ablation study confirms that the label-aware masked language modeling task is crucial, as SentiLARE-EF-LS (without late supervision) underperforms the full SentiLARE model, especially on complex datasets like SST-More.
• The context-aware sentiment attention mechanism successfully captures variable sentiment roles of words across different parts of speech and contexts, outperforming static sentiment lexicons.