[Paper Review] Predicting the Semantic Textual Similarity with Siamese CNN and LSTM

Elvys Linhares Pontes, Stéphane Huet+2 more|arXiv (Cornell University)|Oct 24, 2018

Topic Modeling46 citations

TL;DR

The paper presents a Siamese CNN+LSTM model that uses local word context via CNN and global sentence context via LSTM to predict semantic textual similarity, achieving competitive results on the SICK dataset.

ABSTRACT

Semantic Textual Similarity (STS) is the basis of many applications in Natural Language Processing (NLP). Our system combines convolution and recurrent neural networks to measure the semantic similarity of sentences. It uses a convolution network to take account of the local context of words and an LSTM to consider the global context of sentences. This combination of networks helps to preserve the relevant information of sentences and improves the calculation of the similarity between sentences. Our model has achieved good results and is competitive with the best state-of-the-art systems.

Motivation & Objective

• Motivate STS as a core NLP task across applications like information retrieval and QA.
• Propose a neural architecture that combines CNN-based local context with LSTM-based global sentence representation.
• Demonstrate that local context improves STS predictions over a pure Siamese LSTM.
• Evaluate the model on the SICK dataset and compare against state-of-the-art baselines.

Proposed method

• Use pre-trained word embeddings as general word representations.
• Apply a Siamese CNN to generate local contexts for each word from neighboring words.
• Concatenate the word embedding with its CNN-derived local context for each word.
• Encode the sentence with an LSTM that processes the sequence of enriched word representations.
• Compute sentence similarity using Manhattan distance between final sentence representations and apply post-hoc regression to map to 1–5 scores.

Experimental results

Research questions

• RQ1 Can incorporating local word context via a Siamese CNN improve semantic similarity estimation over a plain Siamese LSTM?
• RQ2 What is the impact of different local-context window sizes on STS performance?
• RQ3 How does the proposed model compare to existing state-of-the-art STS systems on the SICK dataset?

Key findings

• The local-context-enhanced Siamese model improves over the baseline Siamese LSTM in correlated metrics and error when assessed on SICK.
• Short local context (3 words) and longer contexts (7, 9 words) offer varying benefits, with 5-word context providing strong performance.
• The proposed model achieves competitive results relative to several state-of-the-art architectures (e.g., tree-based LSTMs, convolutional nets) on the STS task.
• Table 1 shows that several baselines outperform the public Siamese LSTM, while the local-context variants reach top-tier performance in several metrics.