[Paper Review] Deep Learning for Answer Sentence Selection

Lei Yu, Karl Moritz Hermann+2 more|arXiv (Cornell University)|Dec 4, 2014

Topic Modeling20 references354 citations

TL;DR

This paper proposes a deep learning approach for answer sentence selection using distributed sentence representations via bag-of-words and convolutional neural network (CNN)-based bigram models, trained on pre-trained word embeddings. The method achieves state-of-the-art performance on the TREC QA benchmark without feature engineering or external linguistic resources, outperforming prior methods that rely on hand-crafted features and semantic resources by leveraging semantic matching and simple word co-occurrence features.

ABSTRACT

Answer sentence selection is the task of identifying sentences that contain the answer to a given question. This is an important problem in its own right as well as in the larger context of open domain question answering. We propose a novel approach to solving this task via means of distributed representations, and learn to match questions with answers by considering their semantic encoding. This contrasts prior work on this task, which typically relies on classifiers with large numbers of hand-crafted syntactic and semantic features and various external resources. Our approach does not require any feature engineering nor does it involve specialist linguistic data, making this model easily applicable to a wide range of domains and languages. Experimental results on a standard benchmark dataset from TREC demonstrate that---despite its simplicity---our model matches state of the art performance on the answer sentence selection task.

Motivation & Objective

• To address the challenge of answer sentence selection in open-domain question answering using a neural network-based approach.
• To eliminate reliance on hand-crafted syntactic and semantic features and external linguistic resources such as WordNet.
• To develop a model that generalizes across domains and languages with minimal architectural complexity.
• To improve performance over traditional count-based baselines by incorporating distributed semantic representations.
• To demonstrate that simple neural sentence models can match or exceed state-of-the-art results in answer sentence selection.

Proposed method

• The model uses pre-trained word embeddings to encode questions and candidate answers into dense vector representations.
• A bag-of-words sentence model and a CNN-based bigram model are used to learn distributed sentence representations.
• A supervised matching function is trained to predict relevance between question-answer pairs based on their semantic encoding.
• The approach combines the neural matching signal with a simple, weighted word co-occurrence counter to improve robustness.
• The model is trained end-to-end on the TREC QA dataset without requiring linguistic annotations or domain-specific feature engineering.
• An enhanced version of the model fuses distributional semantics with surface-level word matching to improve performance on rare or out-of-vocabulary words.

Experimental results

Research questions

• RQ1 Can a neural network-based sentence model effectively learn semantic matching between questions and answer sentences without feature engineering?
• RQ2 How does the performance of a distributional sentence model compare to traditional count-based baselines in answer sentence selection?
• RQ3 To what extent can pre-trained word embeddings and simple neural architectures replace complex, hand-coded linguistic features in this task?
• RQ4 Can a model trained on general corpora generalize across domains and languages without retraining or resource adaptation?
• RQ5 Does combining distributional semantics with surface-level word matching improve performance on challenging cases involving numbers or proper nouns?

Key findings

• The proposed model achieves state-of-the-art performance on the TREC answer sentence selection benchmark, matching the best-reported results without using external semantic resources.
• The addition of distributed semantic representations improved both MAP and MRR scores by approximately 10% over the pure co-occurrence count baseline.
• The CNN-based bigram model outperformed the bag-of-words model, demonstrating the benefit of capturing local n-gram patterns in sentence encoding.
• The combined model (bigram + co-occurrence) successfully distinguished correct answers in cases where word counts alone failed, such as when both candidates had identical word overlap with the question.
• The model effectively handled proper nouns and numerical expressions by leveraging semantic similarity (e.g., 'die' and 'killed' being semantically close), even when words were out-of-vocabulary and assigned the UNKNOWN token.
• Despite limitations in capturing complex syntactic or world-knowledge dependencies (e.g., 'group' vs. 'lead singer'), the model significantly outperformed purely lexical baselines.