[Paper Review] RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering

Yingqi Qu, Yuchen Ding+7 more|arXiv (Cornell University)|Oct 16, 2020

Topic ModelingComputer Science34 references73 citations

TL;DR

RocketQA introduces three training strategies—cross-batch negatives, denoised hard negatives, and data augmentation—to improve dense passage retrieval for open-domain QA, achieving strong gains on MSMARCO and Natural Questions and improving end-to-end QA performance.

ABSTRACT

In open-domain question answering, dense passage retrieval has become a new paradigm to retrieve relevant passages for finding answers. Typically, the dual-encoder architecture is adopted to learn dense representations of questions and passages for semantic matching. However, it is difficult to effectively train a dual-encoder due to the challenges including the discrepancy between training and inference, the existence of unlabeled positives and limited training data. To address these challenges, we propose an optimized training approach, called RocketQA, to improving dense passage retrieval. We make three major technical contributions in RocketQA, namely cross-batch negatives, denoised hard negatives and data augmentation. The experiment results show that RocketQA significantly outperforms previous state-of-the-art models on both MSMARCO and Natural Questions. We also conduct extensive experiments to examine the effectiveness of the three strategies in RocketQA. Besides, we demonstrate that the performance of end-to-end QA can be improved based on our RocketQA retriever.

Motivation & Objective

• Address the challenges of training dense dual-encoder retrievers for open-domain QA, including training/inference discrepancy, unlabeled positives, and limited data.
• Propose a practical optimization pipeline to improve retrieval quality and efficiency.
• Demonstrate that the learned retriever enhances end-to-end QA performance.
• Provide analysis of each proposed strategy's contribution to overall gains.

Proposed method

• Use a dual-encoder architecture to learn question and passage embeddings for semantic matching.
• Introduce cross-batch negatives to expand the set of negatives across multiple GPUs during training.
• Apply denoised hard negatives by leveraging a cross-encoder to filter likely false negatives.
• Employ data augmentation by using a cross-encoder to label unlabeled data and distill its signals into the dual-encoder.
• Train the dual-encoder in multi-step RocketQA procedure, including pretraining and pseudo-labeled data incorporation.

Experimental results

Research questions

• RQ1 Can cross-batch negatives reduce the training/inference gap in dual-encoder retrieval for open-domain QA?
• RQ2 Do denoised hard negatives (via a cross-encoder) provide more reliable hard negatives than naive top-k negatives?
• RQ3 Does cross-encoder-based data augmentation improve dual-encoder performance with limited labeled data?
• RQ4 To what extent does RocketQA improve end-to-end QA accuracy when paired with a reader?

Key findings

• RocketQA significantly outperforms prior state-of-the-art models on MSMARCO and Natural Questions for dense passage retrieval.
• Cross-batch negatives provide substantial gains by increasing effective negatives during training across GPUs.
• Denoised hard negatives substantially improve retrieval performance compared with hard negatives sampled without denoising.
• Data augmentation using a cross-encoder to label unlabeled data yields further improvements, indicating benefits of teacher-student distillation for retrievers.
• End-to-end QA performance is improved when using the RocketQA retriever with an extractive reader.