[Paper Review] Improving Passage Retrieval with Zero-Shot Question Generation

Devendra Singh Sachan, Michael Lewis+5 more|arXiv (Cornell University)|Apr 15, 2022

Topic Modeling4 citations

TL;DR

This paper proposes Unsupervised Passage Re-ranking (UPR), a zero-shot re-ranker that improves passage retrieval by scoring passages based on the likelihood of generating the input question from each passage using a pre-trained language model. Without any fine-tuning or labeled data, UPR boosts top-20 retrieval accuracy by 6%-18% over unsupervised retrievers and achieves new state-of-the-art results on open-domain QA benchmarks.

ABSTRACT

We propose a simple and effective re-ranking method for improving passage retrieval in open question answering. The re-ranker re-scores retrieved passages with a zero-shot question generation model, which uses a pre-trained language model to compute the probability of the input question conditioned on a retrieved passage. This approach can be applied on top of any retrieval method (e.g. neural or keyword-based), does not require any domain- or task-specific training (and therefore is expected to generalize better to data distribution shifts), and provides rich cross-attention between query and passage (i.e. it must explain every token in the question). When evaluated on a number of open-domain retrieval datasets, our re-ranker improves strong unsupervised retrieval models by 6%-18% absolute and strong supervised models by up to 12% in terms of top-20 passage retrieval accuracy. We also obtain new state-of-the-art results on full open-domain question answering by simply adding the new re-ranker to existing models with no further changes.

Motivation & Objective

• To improve open-domain passage retrieval without requiring task-specific training data or fine-tuning.
• To develop a re-ranker that enables deep cross-attention between query and passage tokens using only pre-trained language models.
• To achieve state-of-the-art performance on open-domain question answering using only inference with a pre-trained reader and re-ranked passages.
• To demonstrate that an unsupervised pipeline can outperform strong supervised dense retrieval models like DPR.
• To explore the generalization and robustness of zero-shot re-ranking across diverse retrievers, language models, and datasets.

Proposed method

• The re-ranker computes the conditional probability p(question | passage) using a pre-trained language model, treating question generation as a proxy for relevance.
• It applies this scoring in a zero-shot manner, requiring no fine-tuning or labeled examples for the retrieval task.
• The method is compatible with any retrieval system, including sparse (e.g., BM25) and dense (e.g., DPR) retrievers.
• The approach leverages cross-attention between query and passage tokens, forcing the model to account for every token in the question.
• It uses instruction-tuned language models such as T0 to improve performance, with prompts like 'Given the following passage, generate a question that it would answer.'
• Re-ranking is applied post-retrieval on the top-K passages, reordering them based on the question generation likelihood score.

Experimental results

Research questions

• RQ1 Can a zero-shot question generation model effectively re-rank retrieved passages without any task-specific fine-tuning?
• RQ2 Does using a pre-trained language model to compute question likelihood improve retrieval accuracy across diverse datasets and retrievers?
• RQ3 Can an unsupervised re-ranking method outperform supervised dense retrieval models like DPR?
• RQ4 How does the choice of pre-trained language model (e.g., instruction-tuned vs. non-instruction-tuned) affect re-ranking performance?
• RQ5 To what extent does the re-ranker improve end-to-end open-domain question answering when used with a pre-trained reader?

Key findings

• UPR improves top-20 retrieval accuracy by 6%-18% absolute when re-ranking top-1000 passages retrieved by Contriever, an unsupervised retriever.
• On the SQuAD-Open and Entity Questions datasets, UPR outperforms BM25 by 14% and 8% in top-20 accuracy, respectively.
• UPR achieves new state-of-the-art results on open-domain QA by simply re-ranking passages and using a pre-trained reader, improving EM scores by up to 3 points.
• The method improves strong supervised retrievers like DPR by up to 12% in top-20 retrieval accuracy.
• Instruction-tuned models such as T0 perform best as re-rankers, highlighting the importance of prompt engineering in zero-shot settings.
• The performance gains are consistent across different retrievers (sparse and dense) and pre-trained language models, indicating strong generalization.