[Paper Review] Improved Answer Selection with Pre-Trained Word Embeddings

Rishav Chakravarti, Jiří Navrátil+1 more|arXiv (Cornell University)|Aug 14, 2017

Topic Modeling9 references3 citations

TL;DR

This paper proposes an enhanced unsupervised re-ranking method, S-RWMD_Q, that improves answer selection by incorporating term proximity into the Relaxed Word Mover's Distance (RWMD) using pre-trained word embeddings. It demonstrates that combining these embedding-based features with traditional information retrieval techniques significantly boosts performance, achieving results comparable to state-of-the-art deep learning models without requiring labeled training data.

ABSTRACT

This paper evaluates existing and newly proposed answer selection methods based on pre-trained word embeddings. Word embeddings are highly effective in various natural language processing tasks and their integration into traditional information retrieval (IR) systems allows for the capture of semantic relatedness between questions and answers. Empirical results on three publicly available data sets show significant gains over traditional term frequency based approaches in both supervised and unsupervised settings. We show that combining these word embedding features with traditional learning-to-rank techniques can achieve similar performance to state-of-the-art neural networks trained for the answer selection task.

Motivation & Objective

• To evaluate and enhance answer selection methods using pre-trained word embeddings in both supervised and unsupervised settings.
• To address lexical gaps in traditional term-overlap methods by leveraging semantic similarity through word embeddings.
• To propose a novel extension of RWMD that accounts for term proximity in answer text, improving semantic matching.
• To demonstrate that combining traditional IR features with embedding-based features can match or exceed performance of complex neural networks.
• To show that unsupervised embedding-based re-ranking can outperform supervised term-based baselines.

Proposed method

• Proposes S-RWMD_Q, a proximity-aware variant of Relaxed Word Mover’s Distance that evaluates word matches across spans of up to 20 consecutive tokens in the answer text.
• Uses min-max normalized scores from a baseline language model (LM) as a first-pass retrieval method.
• Combines the LM baseline with S-RWMD_Q via simple summation (CombSUM) to improve ranking performance.
• Integrates S-RWMD_Q and other embedding-based features (e.g., MMP_0.7) into a supervised learning-to-rank framework using LambdaMART.
• Employs pre-trained word embeddings (Word2Vec/GloVe) to compute semantic similarity between question and answer terms.
• Applies span-based analysis to capture local co-occurrence patterns, enhancing the robustness of word matching beyond isolated terms.

Experimental results

Research questions

• RQ1 Can a proximity-aware extension of RWMD improve answer selection performance over standard RWMD in unsupervised settings?
• RQ2 Does combining traditional IR features with embedding-based features yield better performance than either alone?
• RQ3 Can unsupervised embedding-based re-ranking outperform supervised term-overlap baselines in answer selection?
• RQ4 How does the performance of the proposed S-RWMD_Q compare to state-of-the-art deep learning models in answer selection?
• RQ5 To what extent do embedding-based features enhance learning-to-rank models when combined with traditional features?

Key findings

• The proposed S-RWMD_Q method significantly outperforms standard RWMD and MMP_0.7 across all three benchmark datasets in both unsupervised and supervised settings.
• Combining LM baseline with S-RWMD_Q (LM+S-RWMD_Q) achieves the best unsupervised performance on all three datasets, demonstrating a statistically significant improvement.
• The supervised learning-to-rank model enhanced with embedding-based features (Embedding-λM) outperforms the base λM model and achieves performance comparable to the AP-CNN neural network model.
• In the smallest dataset, the embedding-enhanced λM model surpasses the AP-CNN model, indicating that feature augmentation can compensate for limited training data.
• The combination of LM and S-RWMD_Q consistently outperforms other hybrid models, including those using MinDist or Spanning-TF-IDF, confirming the value of semantic proximity in embedding space.
• Even without labeled data, the unsupervised S-RWMD_Q method achieves performance that exceeds that of supervised term-overlap baselines, highlighting the effectiveness of semantic matching.