[Paper Review] Semantic Sentence Matching with Densely-connected Recurrent and Co-attentive Information

Seonhoon Kim, In-Ho Kang+1 more|arXiv (Cornell University)|May 29, 2018

Topic Modeling35 references40 citations

TL;DR

The paper introduces DRCN, a densely-connected recurrent and co-attentive network for semantic sentence matching, using autoencoder bottlenecks to control feature growth, and achieves state-of-the-art results on several benchmarks without external knowledge.

ABSTRACT

Sentence matching is widely used in various natural language tasks such as natural language inference, paraphrase identification, and question answering. For these tasks, understanding logical and semantic relationship between two sentences is required but it is yet challenging. Although attention mechanism is useful to capture the semantic relationship and to properly align the elements of two sentences, previous methods of attention mechanism simply use a summation operation which does not retain original features enough. Inspired by DenseNet, a densely connected convolutional network, we propose a densely-connected co-attentive recurrent neural network, each layer of which uses concatenated information of attentive features as well as hidden features of all the preceding recurrent layers. It enables preserving the original and the co-attentive feature information from the bottommost word embedding layer to the uppermost recurrent layer. To alleviate the problem of an ever-increasing size of feature vectors due to dense concatenation operations, we also propose to use an autoencoder after dense concatenation. We evaluate our proposed architecture on highly competitive benchmark datasets related to sentence matching. Experimental results show that our architecture, which retains recurrent and attentive features, achieves state-of-the-art performances for most of the tasks.

Motivation & Objective

• Motivate improved semantic understanding between two sentences for tasks like natural language inference, paraphrase identification, and answer selection.
• Propose a deeper, densely-connected recurrent architecture that preserves both recurrent and co-attentive features across layers.
• Incorporate co-attentive information via concatenation to retain rich interactions between sentences.
• Control feature growth with an autoencoder bottleneck to maintain tractable model size while preserving information.

Proposed method

• Build word representations by concatenating trainable and fixed word embeddings, character-level CNN features, and an exact-match flag.
• Stack BiLSTM-based RNN layers with dense connections that concatenate outputs from all preceding layers.
• Compute co-attentive context vectors via a cosine similarity-based attention between sentence representations and fuse them through dense concatenation.
• Apply step-wise max-pooling over the densely connected recurrent and co-attentive features to obtain fixed-length sentence representations.
• Combine sentence representations with interactions (p, q, p+q, p−q, |p−q|) and pass through two dense layers for classification, trained with cross-entropy and autoencoder reconstruction loss.
• Use an autoencoder as a bottleneck to compress the dense feature set and regularize training.

Experimental results

Research questions

• RQ1 How can densely-connected recurrent and co-attentive information improve semantic sentence matching over deeper architectures?
• RQ2 Does preserving both original and co-attentive features through dense concatenation outperform residual connections in deep RNNs?
• RQ3 Can an autoencoder bottleneck effectively regulate feature growth while maintaining or improving performance on multiple sentence-pair tasks?
• RQ4 What is the impact of mixing trainable and fixed word embeddings on overall performance?

Key findings

• DRCN achieves competitive or state-of-the-art accuracy on SNLI and MultiNLI without external contextualized word representations.
• Ensemble DR CN achieves 90.1% on SNLI in the reported results, surpassing several baselines without external knowledge.
• DRCN delivers strong results on Quora paraphrase identification (MAP 0.804, MRR 0.862; ensemble 0.830, 0.908) and sets new state-of-the-art on TrecQA and SelQA for answer sentence selection.
• Ablation studies show that dense connections and attentive information are crucial for performance, with attention contributing significantly to semantic matching.
• Incorporating ELMo or contextualized embeddings further improves performance on MultiNLI, indicating complementary gains with DRCN.