[Paper Review] A Structured Self-attentive Sentence Embedding

Zhouhan Lin, Minwei Feng+5 more|arXiv (Cornell University)|Mar 9, 2017

Topic Modeling1,463 citations

TL;DR

The paper introduces a self-attentive, 2-D matrix sentence embedding that attends to multiple sentence parts via a structured attention mechanism, enabling visualization and improved performance across author profiling, sentiment analysis, and textual entailment.

ABSTRACT

This paper proposes a new model for extracting an interpretable sentence embedding by introducing self-attention. Instead of using a vector, we use a 2-D matrix to represent the embedding, with each row of the matrix attending on a different part of the sentence. We also propose a self-attention mechanism and a special regularization term for the model. As a side effect, the embedding comes with an easy way of visualizing what specific parts of the sentence are encoded into the embedding. We evaluate our model on 3 different tasks: author profiling, sentiment classification, and textual entailment. Results show that our model yields a significant performance gain compared to other sentence embedding methods in all of the 3 tasks.

Motivation & Objective

• Develop an interpretable sentence embedding using self-attention implemented as a 2-D matrix.
• Enable multiple attention hops to capture different semantic aspects of a sentence.
• Provide a regularization term to reduce redundancy among attention hops and improve interpretability.
• Demonstrate the approach on author profiling, sentiment analysis, and textual entailment tasks.
• Offer visualization techniques to interpret which sentence parts contribute to the embedding.

Proposed method

• Bidirectional LSTM to produce hidden states for each token.
• Compute a set of attention weight vectors via a two-layer MLP (softmax over hops) to form an annotation matrix A.
• Construct a fixed-size embedding M = AH, where H is the sequence of LSTM hidden states and A encodes r attention hops.
• Extend to multiple hops (r rows) in the embedding to capture diverse semantic aspects.
• Introduce a penalization term P = ||AA^T - I||_F^2 to encourage diversity and focus among attention hops.
• Optionally visualize per-hop and aggregate attention to interpret which words influence the embedding.

Experimental results

Research questions

• RQ1 Can self-attentive, matrix-based sentence embeddings outperform traditional vector embeddings on diverse NLP tasks?
• RQ2 Does a diversity-encouraging penalty improve interpretability and model performance?
• RQ3 How does multi-hop attention (varying r) affect performance across tasks of increasing sentence length?
• RQ4 Can the model provide intuitive visualizations that align with human interpretation of sentence semantics?

Key findings

• On Yelp sentiment classification, the proposed model achieves 64.21% accuracy versus 61.99% (BiLSTM+Max) and 62.05% (CNN+Max).
• On the Age author-profiling task, the model achieves 80.45% accuracy versus 77.40% (BiLSTM+Max) and 78.15% (CNN+Max).
• The penalization term improves performance on Yelp and Age, yielding 1.0 penalty results (64.21% and 80.45%) vs 0.0 penalty (61.74% and 79.27%).
• For SNLI, the proposed method achieves competitive results (84.4% test accuracy) against other sentence-embedding-based models, often close to state-of-the-art methods.