[Paper Review] Neural Passage Retrieval with Improved Negative Contrast

Jing Lü, Gustavo Hernández Ábrego+3 more|arXiv (Cornell University)|Oct 23, 2020

Topic ModelingComputer Science29 references21 citations

TL;DR

This paper proposes improved negative sampling strategies—based on semantic similarity, lexical overlap, and heuristics—for training dual encoder models in neural passage retrieval. By applying these hard negatives in both pre-training and fine-tuning stages, the method significantly improves contrast between relevant and irrelevant passages, achieving state-of-the-art performance on Open Domain QA benchmarks with gains of 0.8–2.9 points in accuracy.

ABSTRACT

In this paper we explore the effects of negative sampling in dual encoder models used to retrieve passages for automatic question answering. We explore four negative sampling strategies that complement the straightforward random sampling of negatives, typically used to train dual encoder models. Out of the four strategies, three are based on retrieval and one on heuristics. Our retrieval-based strategies are based on the semantic similarity and the lexical overlap between questions and passages. We train the dual encoder models in two stages: pre-training with synthetic data and fine tuning with domain-specific data. We apply negative sampling to both stages. The approach is evaluated in two passage retrieval tasks. Even though it is not evident that there is one single sampling strategy that works best in all the tasks, it is clear that our strategies contribute to improving the contrast between the response and all the other passages. Furthermore, mixing the negatives from different strategies achieve performance on par with the best performing strategy in all tasks. Our results establish a new state-of-the-art level of performance on two of the open-domain question answering datasets that we evaluated.

Motivation & Objective

• To systematically evaluate the impact of hard negative sampling on dual encoder models for passage retrieval.
• To explore whether hard negatives from retrieval-based and heuristic-based strategies improve contrast between relevant and irrelevant passages.
• To investigate the effectiveness of integrating hard negatives in both pre-training and fine-tuning stages using synthetic and gold data.
• To determine whether ensembling models trained with different negative sampling strategies yields superior performance.
• To establish a new state-of-the-art in open-domain question answering by combining synthetic pre-training with hard negative sampling.

Proposed method

• Uses a two-stage training framework: pre-training on synthetic question-passage pairs generated via a question generator model, followed by fine-tuning on gold-labeled data.
• Applies four distinct hard negative sampling strategies: coarse/fine semantic similarity, BM25-based lexical overlap, and heuristic-based context negatives.
• Integrates hard negatives into both pre-training and fine-tuning stages to enhance contrast between positive and negative passages.
• Employs BERT-based dual encoders to independently embed questions and passages into a shared dense vector space.
• Employs contrastive learning with hard negatives to optimize the model to push relevant passages closer and irrelevant ones farther in embedding space.
• Ensembles models trained with different negative sampling strategies to leverage complementary strengths and achieve SOTA performance.

Experimental results

Research questions

• RQ1 How do different hard negative sampling strategies affect the performance of dual encoder models in passage retrieval?
• RQ2 Does incorporating hard negatives in the pre-training stage improve model generalization compared to random negatives?
• RQ3 Which negative sampling strategy yields the best performance across diverse question types and domains (e.g., NQ vs. SQuAD)?
• RQ4 Can combining models trained with multiple negative sampling strategies through ensembling outperform individual strategies?
• RQ5 To what extent does synthetic pre-training with hard negatives reduce reliance on expensive gold-labeled data in retrieval tasks?

Key findings

• All four hard negative sampling strategies—coarse/fine semantic similarity, BM25-based, and heuristic context negatives—significantly improved model performance over random negatives in both pre-training and fine-tuning stages.
• The use of hard negatives in both pre-training and fine-tuning stages led to consistent performance gains, with the best results achieved when hard negatives were used throughout both stages.
• On the Natural Questions (NQ) dataset, context-based hard negatives yielded the best performance, while semantic similarity-based negatives (both coarse and fine) outperformed others on SQuAD.
• Ensembling models trained with different negative sampling strategies achieved state-of-the-art performance across all evaluated tasks, with improvements of 0.8–2.9 points in accuracy over prior work.
• Removing hard negatives from the pre-training stage caused a significant drop in performance, demonstrating the value of hard negatives even in synthetic pre-training.
• Fine-tuning directly from a BERT checkpoint (without pre-training) led to a substantial performance drop, confirming the effectiveness of synthetic pre-training with hard negatives.