[Paper Review] LSTM-based Deep Learning Models for Non-factoid Answer Selection

Ming Tan, Cícero dos Santos+2 more|arXiv (Cornell University)|Nov 12, 2015

Topic ModelingComputer Science24 references403 citations

TL;DR

This paper proposes a bidirectional LSTM-based deep learning framework for non-factoid answer selection that eliminates reliance on handcrafted features or linguistic tools. By combining biLSTM with CNN and an attention mechanism for enhanced context-aware representations, the model achieves state-of-the-art performance on both TREC-QA and InsuranceQA datasets, outperforming strong baselines by up to 4% in MRR on TREC-QA.

ABSTRACT

In this paper, we apply a general deep learning (DL) framework for the answer selection task, which does not depend on manually defined features or linguistic tools. The basic framework is to build the embeddings of questions and answers based on bidirectional long short-term memory (biLSTM) models, and measure their closeness by cosine similarity. We further extend this basic model in two directions. One direction is to define a more composite representation for questions and answers by combining convolutional neural network with the basic framework. The other direction is to utilize a simple but efficient attention mechanism in order to generate the answer representation according to the question context. Several variations of models are provided. The models are examined by two datasets, including TREC-QA and InsuranceQA. Experimental results demonstrate that the proposed models substantially outperform several strong baselines.

Motivation & Objective

• To develop a deep learning framework for answer selection that does not require manual feature engineering or linguistic tools.
• To improve answer representation by combining biLSTM with convolutional neural networks for richer local feature extraction.
• To enhance matching between questions and answers using an attention mechanism that focuses on relevant question context during answer encoding.
• To evaluate the proposed models on two diverse non-factoid QA datasets: TREC-QA and InsuranceQA.
• To demonstrate that end-to-end learning with attention and hierarchical representations outperforms traditional feature-based and CNN-only baselines.

Proposed method

• The model uses bidirectional LSTMs to encode both questions and answer candidates into dense vector representations.
• A max-pooling layer is applied to the final hidden states of the biLSTM to produce a fixed-size representation for each sequence.
• A convolutional neural network (CNN) is stacked on top of the biLSTM outputs to capture local n-gram features and improve representation quality.
• An attention mechanism is introduced to generate answer representations conditioned on the question context, allowing dynamic focus on relevant parts of the answer.
• Cosine similarity is used to measure the matching score between question and answer embeddings.
• The model is trained end-to-end using stochastic gradient descent with cross-entropy loss, optimizing for ranking the correct answer highest.

Experimental results

Research questions

• RQ1 Can a deep learning framework based solely on biLSTM and attention achieve superior answer selection performance without relying on linguistic features or external resources?
• RQ2 How does combining CNN with biLSTM improve representation learning for questions and answers in non-factoid QA?
• RQ3 To what extent does incorporating an attention mechanism that aligns answers with question context enhance matching accuracy?
• RQ4 Does the proposed model generalize across different domains, as evidenced by performance on both TREC-QA and InsuranceQA?
• RQ5 How does the model compare to strong baselines, including CNN-only and feature-engineered models, in terms of MAP and MRR?

Key findings

• The proposed model with biLSTM, CNN, and attention (Model E) achieved the highest performance on TREC-QA, with a MAP of 72.79% and MRR of 82.40%, outperforming the best baseline by 1.45% in MAP and 2.9% in MRR.
• Model D, which combines biLSTM/CNN with attention, achieved a MRR of 83.22%, representing a 4% improvement over the baseline of Wang & Nyberg (2015) on MRR.
• The attention mechanism alone (Model B) improved MRR by approximately 2% over the basic QA-LSTM model, demonstrating its effectiveness in focusing on relevant answer parts.
• The CNN-enhanced model (Model C) improved both MAP and MRR by over 2% compared to the basic QA-LSTM model, showing that local feature learning enhances representation.
• The model without attention (Model A) achieved a MAP of 68.19% and MRR of 76.52%, indicating that even the basic biLSTM framework outperforms earlier feature-based methods.
• The results on the InsuranceQA dataset showed substantial improvements over two non-DL baselines and a strong DL baseline based on CNN, confirming the framework’s generalization across domains.