[Paper Review] Recurrent Neural Network-Based Sentence Encoder with Gated Attention for Natural Language Inference

Qian Chen, Xiaodan Zhu+4 more|arXiv (Cornell University)|Aug 4, 2017

Topic Modeling10 references22 citations

TL;DR

This paper proposes a recurrent neural network-based sentence encoder with gated attention for natural language inference, achieving state-of-the-art performance on both the RepEval 2017 and SNLI benchmarks. By integrating intra-sentence gated attention into a stacked bidirectional LSTM architecture with character-level and word-level embeddings, the model attains 74.9% accuracy on both in-domain and cross-domain test sets in RepEval 2017 and 85.5% on SNLI without cross-sentence attention, outperforming prior sentence-encoder-only models.

ABSTRACT

The RepEval 2017 Shared Task aims to evaluate natural language understanding models for sentence representation, in which a sentence is represented as a fixed-length vector with neural networks and the quality of the representation is tested with a natural language inference task. This paper describes our system (alpha) that is ranked among the top in the Shared Task, on both the in-domain test set (obtaining a 74.9% accuracy) and on the cross-domain test set (also attaining a 74.9% accuracy), demonstrating that the model generalizes well to the cross-domain data. Our model is equipped with intra-sentence gated-attention composition which helps achieve a better performance. In addition to submitting our model to the Shared Task, we have also tested it on the Stanford Natural Language Inference (SNLI) dataset. We obtain an accuracy of 85.5%, which is the best reported result on SNLI when cross-sentence attention is not allowed, the same condition enforced in RepEval 2017.

Motivation & Objective

• To develop a sentence encoder model that achieves high performance in natural language inference without relying on cross-sentence attention mechanisms.
• To investigate whether the full potential of sentence-encoder-based models has been realized, especially in comparison to attention-based models.
• To improve sentence representation quality through a novel intra-sentence gated-attention mechanism within a bidirectional LSTM framework.
• To evaluate generalization performance across in-domain and cross-domain NLI datasets.
• To establish a strong baseline for sentence encoding in NLI that does not require attention between premise and hypothesis.

Proposed method

• Employs a hybrid word representation combining character-level CNN embeddings and pre-trained GloVe word embeddings for improved out-of-vocabulary and morphological generalization.
• Uses stacked bidirectional LSTMs with shortcut connections to encode premise and hypothesis sequences into contextual hidden states.
• Introduces a novel intra-sentence gated-attention mechanism that selectively focuses on important words within each sentence using input, forget, and output gates.
• Applies element-wise product and absolute difference operations on encoded premise and hypothesis representations to capture interaction patterns.
• Uses a final classifier head with a softmax layer to predict entailment, contradiction, or neutral labels based on the combined sentence representations.
• Applies model ensembling by averaging five models trained with different random initializations to improve robustness and performance.

Experimental results

Research questions

• RQ1 Can a sentence-encoder-only model achieve state-of-the-art performance in natural language inference without using cross-sentence attention?
• RQ2 How effective is intra-sentence gated attention in improving sentence representation quality for NLI tasks?
• RQ3 To what extent do character-level composition and pre-trained word embeddings contribute to model generalization on out-of-domain data?
• RQ4 Does the proposed model generalize well to cross-domain NLI benchmarks compared to existing sentence-encoder models?
• RQ5 What is the relative contribution of different architectural components (e.g., gated attention, character CNN, word embeddings) to overall performance?

Key findings

• The proposed model achieved 74.9% accuracy on both the in-domain and cross-domain test sets in the RepEval 2017 Shared Task, ranking first overall.
• When applied to the SNLI dataset without cross-sentence attention, the model achieved 85.5% accuracy, the highest reported result among sentence-encoder-only models.
• Removing the gated-attention mechanism reduced performance to 72.8% (in-domain) and 73.6% (cross-domain), demonstrating its critical role in performance.
• The ablation study showed that word-level embeddings contributed most significantly to performance, as removing them dropped accuracy to 65.6% and 66.0%.
• The model with character-level composition and word-level embeddings outperformed models using only one of the two, indicating complementary benefits.
• Ensembling five independently initialized models improved performance to 74.9% on both test sets, confirming the effectiveness of model averaging.