[Paper Review] Attentional Encoder Network for Targeted Sentiment Classification

Youwei Song, Jiahai Wang+3 more|arXiv (Cornell University)|Feb 25, 2019

Topic Modeling26 references197 citations

TL;DR

This paper introduces the Attentional Encoder Network (AEN), a non-recurrent, attention-based model for targeted sentiment classification, augmented with label smoothing, and shows state-of-the-art results with BERT integration.

ABSTRACT

Targeted sentiment classification aims at determining the sentimental tendency towards specific targets. Most of the previous approaches model context and target words with RNN and attention. However, RNNs are difficult to parallelize and truncated backpropagation through time brings difficulty in remembering long-term patterns. To address this issue, this paper proposes an Attentional Encoder Network (AEN) which eschews recurrence and employs attention based encoders for the modeling between context and target. We raise the label unreliability issue and introduce label smoothing regularization. We also apply pre-trained BERT to this task and obtain new state-of-the-art results. Experiments and analysis demonstrate the effectiveness and lightweight of our model.

Motivation & Objective

• Motivate targeted sentiment classification and address limitations of RNN-based encoders.
• Propose a lightweight attentional encoder that models target-context interactions without recurrence.
• Introduce label smoothing regularization to handle label unreliability in neutral sentiment.
• Demonstrate that integrating BERT improves performance and provide model size analysis.

Proposed method

• Embed words using GloVe or BERT (AEN-GloVe vs AEN-BERT).
• Use a parallel attentional encoder with intra-MHA (context self-attention) and inter-MHA (context-to-target), followed by a target-specific MHA.
• Apply a point-wise convolution transformation (PCT) to refine representations.
• Aggregate via average pooling and a dense layer for classification with softmax.
• Incorporate label smoothing regularization (LSR) to mitigate neutral label unreliability; optimize with cross-entropy + LSR + L2 regularization.

Experimental results

Research questions

• RQ1 Can a non-recurrent attentional encoder effectively model interactions between target and context for targeted sentiment classification?
• RQ2 Does label smoothing improve robustness to unreliable neutral labels?
• RQ3 How does a BERT-enhanced AEN compare to standard BERT baselines and RNN-based models in targeted sentiment tasks?
• RQ4 What are the trade-offs in model size and computational efficiency compared to recurrent architectures?

Key findings

• AEN-BERT and AEN-GloVe achieve competitive or superior accuracy and Macro-F1 across Twitter, Restaurant, and Laptop datasets.
• AEN-GloVe with the proposed components outperforms ablations lacking PCT, MHA, or LSR, demonstrating the importance of each module.
• BERT-SPC and AEN-BERT achieve substantial gains, with AEN-BERT surpassing BERT-SPC, highlighting the benefit of task-tailored architectures on top of pretrained models.
• AEN-GloVe is a lightweight alternative to recurrent models, with smaller parameter counts and memory usage.
• Label smoothing (LSR) significantly improves performance, especially due to neutral label unreliability.
• BERT-based variants show strong gains indicating the value of task-specific downstream architecture.