[Paper Review] Modeling Relational Information in Question-Answer Pairs with Convolutional Neural Networks

Aliaksei Severyn, Alessandro Moschitti|arXiv (Cornell University)|Apr 5, 2016

Topic Modeling24 references45 citations

TL;DR

This paper proposes a convolutional neural network (CNN) model that enhances question-answer sentence selection by injecting relational information—specifically, word overlaps—directly into word embeddings via additional trainable dimensions. By jointly learning sentence representations and relational features end-to-end, the model achieves state-of-the-art performance on the WikiQA benchmark, attaining an MRR of 71.07 and MAP of 69.51.

ABSTRACT

In this paper, we propose convolutional neural networks for learning an optimal representation of question and answer sentences. Their main aspect is the use of relational information given by the matches between words from the two members of the pair. The matches are encoded as embeddings with additional parameters (dimensions), which are tuned by the network. These allows for better capturing interactions between questions and answers, resulting in a significant boost in accuracy. We test our models on two widely used answer sentence selection benchmarks. The results clearly show the effectiveness of our relational information, which allows our relatively simple network to approach the state of the art.

Motivation & Objective

• To improve answer sentence selection by modeling relational information between questions and answers beyond simple lexical matching.
• To develop a deep learning architecture that jointly learns sentence representations and relational features in an end-to-end manner.
• To replace heuristic feature engineering (e.g., word overlap counts) with learnable, embedded relational features for better generalization.
• To achieve state-of-the-art performance on answer sentence selection benchmarks using a simpler, yet more expressive, architecture.

Proposed method

• Uses two parallel convolutional neural networks (CNNs) to encode questions and answers into dense vector representations.
• Introduces additional embedding dimensions to represent word matches between question and answer sentences, which are learned during training.
• Combines the intermediate sentence representations and their similarity score into a richer joint representation for final scoring.
• Employs a final scoring function that integrates both the similarity score and the individual sentence representations for reranking.
• Trains the entire model end-to-end, allowing joint optimization of word embeddings, relational features, and classification head.
• Uses only pre-trained word embeddings as initialization, but allows end-to-end fine-tuning for task-specific optimization.

Experimental results

Research questions

• RQ1 Can relational information between question and answer sentences be effectively modeled using learnable embedding dimensions?
• RQ2 Does injecting word overlap information directly into embeddings improve answer sentence selection performance compared to post-hoc feature engineering?
• RQ3 Can a simpler CNN architecture with richer relational modeling outperform more complex models that rely on external features?
• RQ4 How does the integration of intermediate sentence representations and similarity scores affect final ranking performance?
• RQ5 Can end-to-end training of relational features lead to state-of-the-art results on standard benchmarks like WikiQA?

Key findings

• The proposed model achieves state-of-the-art performance on the WikiQA benchmark, with an MRR of 71.07 and MAP of 69.51.
• The model outperforms the ABCNN model and surpasses NASM c, which previously outperformed it on TREC13, confirming its strong generalization on larger datasets.
• The difference between CNN and CNN R (with relational features) is significant, demonstrating the value of relational modeling in improving accuracy.
• The model performs competitively on TREC13, though the small dataset limits definitive ranking claims, highlighting the importance of dataset size for reliable evaluation.
• End-to-end training with learnable relational embeddings leads to better performance than models that combine CNN outputs with external logistic regression features.
• The ablation study confirms that relational features are crucial, as removing them reduces performance from MAP .7654 to .7186 and MRR from .8186 to .7828.