[Paper Review] Reasoning about Entailment with Neural Attention

Tim Rocktäschel, Edward Grefenstette+3 more|arXiv (Cornell University)|Sep 22, 2015

Natural Language Processing Techniques406 citations

TL;DR

This paper proposes a novel end-to-end differentiable neural network for textual entailment that combines bidirectional LSTMs with word-by-word attention to reason over semantic relationships between premise and hypothesis. By conditioning the hypothesis LSTM on the premise's final cell state and attending over all premise hidden states, the model achieves 83.5% accuracy on the SNLI dataset—surpassing both a strong baseline LSTM and a lexical similarity classifier—demonstrating state-of-the-art performance and interpretability through attention visualization.

ABSTRACT

While most approaches to automatically recognizing entailment relations have used classifiers employing hand engineered features derived from complex natural language processing pipelines, in practice their performance has been only slightly better than bag-of-word pair classifiers using only lexical similarity. The only attempt so far to build an end-to-end differentiable neural network for entailment failed to outperform such a simple similarity classifier. In this paper, we propose a neural model that reads two sentences to determine entailment using long short-term memory units. We extend this model with a word-by-word neural attention mechanism that encourages reasoning over entailments of pairs of words and phrases. Furthermore, we present a qualitative analysis of attention weights produced by this model, demonstrating such reasoning capabilities. On a large entailment dataset this model outperforms the previous best neural model and a classifier with engineered features by a substantial margin. It is the first generic end-to-end differentiable system that achieves state-of-the-art accuracy on a textual entailment dataset.

Motivation & Objective

• To develop a generic, end-to-end differentiable neural model for textual entailment without reliance on hand-engineered linguistic features.
• To improve upon existing neural models that encode premise and hypothesis independently by introducing conditional encoding based on the premise's hidden state.
• To enhance reasoning over semantic relationships between sentence pairs through a word-by-word attention mechanism over the premise.
• To provide qualitative interpretability of attention patterns to validate the model's ability to capture entailment, contradiction, and paraphrase relationships.
• To achieve state-of-the-art performance on the Stanford Natural Language Inference (SNLI) dataset using a general sequence-to-sequence architecture.

Proposed method

• Uses bidirectional LSTMs to encode the premise and hypothesis separately, with the hypothesis LSTM conditioned on the final cell state of the premise LSTM.
• Applies a soft attention mechanism that computes attention weights over all hidden states of the premise to focus on relevant words and phrases when processing the hypothesis.
• Employs word-by-word attention by computing attention scores between each word in the hypothesis and every hidden state of the premise, enabling fine-grained alignment.
• Uses a non-linear projection of the final hidden state of the hypothesis LSTM to a softmax classifier for predicting entailment, neutral, or contradiction classes.
• Trains the entire model end-to-end using cross-entropy loss, with fixed word2vec embeddings for in-vocabulary words and learned embeddings for out-of-vocabulary words.
• Enables two-way attention by allowing the model to attend over the hypothesis when processing the premise, though this does not improve performance.

Experimental results

Research questions

• RQ1 Can a differentiable, end-to-end neural network with attention mechanisms outperform traditional feature-based classifiers in textual entailment?
• RQ2 Does conditioning the hypothesis LSTM on the premise's hidden state improve reasoning over semantic relationships between sentence pairs?
• RQ3 Can word-by-word attention over the premise's hidden states enhance the model's ability to detect entailment, contradiction, or paraphrase at the word and phrase level?
• RQ4 How do attention weights reflect meaningful semantic relationships, such as synonymy, contradiction, or paraphrase, in real examples?
• RQ5 Does two-way attention (attending over both premise and hypothesis) improve performance, or does asymmetry in entailment limit its benefit?

Key findings

• The proposed model achieves 83.5% accuracy on the SNLI dataset, setting a new state-of-the-art performance for neural models on this benchmark.
• The model outperforms a strong baseline LSTM (80.9%) by 2.6 percentage points and a lexical similarity classifier by 2.7 percentage points.
• Word-by-word attention provides a 1.2 percentage point improvement over attending only on the final hidden state of the premise.
• Qualitative analysis shows the model correctly attends to semantically coherent words (e.g., 'riding' and 'rides'), contradictory terms (e.g., 'blue' vs. 'pink'), and paraphrases (e.g., 'garbage can' and 'trashcan').
• The model effectively ignores uninformative phrases such as 'which also has a rope leading out of it' when determining entailment, indicating robustness to noise.
• Attention patterns reveal that the model can resolve one-to-many relationships (e.g., 'kids' to 'boy' and 'girl') and use common-sense knowledge (e.g., 'snow' implies 'outside', 'mother' implies 'adult').