[Paper Review] A Replication Study of Dense Passage Retriever

Xueguang Ma, Kai Sun+2 more|arXiv (Cornell University)|Apr 12, 2021

Topic Modeling12 references27 citations

TL;DR

This paper reproduces the Dense Passage Retriever (DPR) and shows that BM25 and dense–sparse hybrids can outperform original DPR claims, and adds end-to-end QA improvements by combining retriever signals and revised answer span scoring.

ABSTRACT

Text retrieval using learned dense representations has recently emerged as a promising alternative to "traditional" text retrieval using sparse bag-of-words representations. One recent work that has garnered much attention is the dense passage retriever (DPR) technique proposed by Karpukhin et al. (2020) for end-to-end open-domain question answering. We present a replication study of this work, starting with model checkpoints provided by the authors, but otherwise from an independent implementation in our group's Pyserini IR toolkit and PyGaggle neural text ranking library. Although our experimental results largely verify the claims of the original paper, we arrived at two important additional findings that contribute to a better understanding of DPR: First, it appears that the original authors under-report the effectiveness of the BM25 baseline and hence also dense--sparse hybrid retrieval results. Second, by incorporating evidence from the retriever and an improved answer span scoring technique, we are able to improve end-to-end question answering effectiveness using exactly the same models as in the original work.

Motivation & Objective

• Assess the reproducibility of the original DPR results using independent implementation and author-provided checkpoints.
• Evaluate dense retrieval, BM25 baselines, and dense–sparse hybrid retrieval across standard QA datasets.
• Explore enhancements in end-to-end QA by integrating retriever signals and improved answer span scoring.
• Examine how different evidence fusion strategies affect final answer extraction.

Proposed method

• Replicate the retriever–reader architecture of DPR using independent Pyserini and PyGaggle implementations, starting from author-provided checkpoints.
• Compare dense retrieval, BM25 sparse retrieval, and hybrids on five QA datasets (NQ, TriviaQA, WQ, CuratedTREC, SQuAD).
• Experiment with different fusion strategies by combining retriever scores with reader scores in end-to-end QA.
• Evaluate two answer span scoring methods: the original DPR technique and Mao et al. (2020) – with and without retriever score fusion.
• Tune hyperparameters (alpha, beta, gamma) via grid search on development data; perform significance testing (paired t-tests with Bonferroni correction).
• Provide replication code within Pyserini (retrieval) and PyGaggle (neural ranking) toolkits.

Experimental results

Research questions

• RQ1 Does the original DPR performance hold under an independent implementation and checkpoint usage?
• RQ2 What is the relative strength of BM25 vs. dense DPR, and do dense–sparse hybrids provide consistent gains?
• RQ3 Can end-to-end QA performance be improved by fusing retriever signals and adopting alternative answer span scoring methods?
• RQ4 How do retrieval choices (k) and evidence fusion impact end-to-end QA accuracy across datasets?

Key findings

• Dense retrieval results closely match the original DPR findings, supporting the effectiveness of DPR.
• BM25 baselines in replication outperform the BM25 figures reported in the original paper, with larger gains at top-20 (≈7-point) than top-100 (≈5-point).
• Dense–sparse hybrids outperform BM25 alone and often improve over dense retrieval, with statistically significant gains across most datasets.
• Combining retriever scores with reader scores and using Mao et al. (2020) span scoring yields about a three-point end-to-end EM improvement over the baseline replication.
• End-to-end QA performance can improve further by using retriever signals for final evidence fusion and applying the Mao et al. span scoring technique with retriever fusion, achieving statistically significant gains on NQ and TriviaQA.