[Paper Review] Large Language Models are Strong Zero-Shot Retriever

Tao Shen, Guodong Long+4 more|arXiv (Cornell University)|Apr 27, 2023

Topic Modeling4 citations

TL;DR

This paper proposes LameR, a zero-shot retrieval method that uses a large language model (LLM) to augment queries with in-domain candidate answers, significantly improving retrieval performance without any fine-tuned retrievers. By combining LLM-based query augmentation with a non-parametric BM25 retriever, LameR achieves state-of-the-art results on benchmark datasets, outperforming both zero-shot and few-shot baselines.

ABSTRACT

In this work, we propose a simple method that applies a large language model (LLM) to large-scale retrieval in zero-shot scenarios. Our method, the Language language model as Retriever (LameR), is built upon no other neural models but an LLM, while breaking brute-force combinations of retrievers with LLMs and lifting the performance of zero-shot retrieval to be very competitive on benchmark datasets. Essentially, we propose to augment a query with its potential answers by prompting LLMs with a composition of the query and the query's in-domain candidates. The candidates, regardless of correct or wrong, are obtained by a vanilla retrieval procedure on the target collection. As a part of the prompts, they are likely to help LLM generate more precise answers by pattern imitation or candidate summarization. Even if all the candidates are wrong, the prompts at least make LLM aware of in-collection patterns and genres. Moreover, due to the low performance of a self-supervised retriever, the LLM-based query augmentation becomes less effective as the retriever bottlenecks the whole pipeline. Therefore, we propose to leverage a non-parametric lexicon-based method (e.g., BM25) as the retrieval module to capture query-document overlap in a literal fashion. As such, LameR makes the retrieval procedure transparent to the LLM, thus circumventing the performance bottleneck.

Motivation & Objective

• To address the performance bottleneck in zero-shot retrieval caused by weak self-supervised retrievers.
• To enable strong zero-shot retrieval without requiring in-domain annotated query-document pairs.
• To improve LLM-based query augmentation by incorporating in-domain candidate documents as context.
• To demonstrate that a non-parametric retriever like BM25 can outperform learned retrievers when combined with LLM-augmented queries.
• To establish a transparent, end-to-end language-model-driven retrieval pipeline that avoids model-specific embedding bottlenecks.

Proposed method

• Augment each query with top candidate passages retrieved via BM25, forming a prompt that includes the query and its candidate answers.
• Use the LLM to generate refined or new answers based on the query and candidate context, leveraging pattern imitation and summarization.
• Treat the LLM’s output as a query augmentation, replacing the original query in a second-stage retrieval with BM25.
• Use a non-parametric, lexicon-based retriever (BM25) instead of a parametric, self-supervised retriever to avoid performance bottlenecks.
• Ensure transparency by relying on literal lexical overlap between augmented queries and documents, avoiding latent space mismatches.
• Apply the method end-to-end: BM25 retrieval → LLM-based query augmentation → second-stage BM25 retrieval.

Figure 1: nDCG@10 on DL19 for query augmentation w/ LLMs.

Experimental results

Research questions

• RQ1 Can LLMs significantly improve zero-shot retrieval performance when provided with in-domain candidate answers as context?
• RQ2 Does replacing a self-supervised retriever with a non-parametric BM25 retriever improve the overall retrieval pipeline when combined with LLM-based query augmentation?
• RQ3 How does LameR compare to existing zero-shot and few-shot retrieval methods in terms of effectiveness and efficiency?
• RQ4 Can LLM-augmented queries achieve state-of-the-art performance without any in-domain fine-tuning or labeled data?
• RQ5 Does the effectiveness of LLM-based query augmentation remain consistent across different retrieval backbones, such as BM25 and dense retrievers?

Key findings

• LameR achieves 69.1 nDCG@10 on TREC DL19 and 64.8 nDCG@10 on DL20, outperforming all zero-shot competitors and surpassing the best fully-supervised retriever (E5 base) on DL20.
• LameR with SimLM dense retriever achieves 76.5 nDCG@10 on DL19 and 75.8 nDCG@10 on DL20, marking a +5.1 and +6.1 improvement over the base retriever, respectively.
• LameR improves BM25 by +18.5 nDCG@10 on DL19 and +16.8 on DL20, significantly outperforming HyDE and Q2D, even when those methods use few-shot demonstrations.
• LameR maintains consistent performance gains across different retrieval backbones, including BM25 and dense retrievers like Contriever and SimLM, indicating robustness to the retrieval model choice.
• LameR reduces retrieval latency and index size compared to HyDE with dense retrievers, due to the efficiency of BM25, while achieving higher effectiveness.
• The method is robust to candidate quality: even with incorrect candidates, the LLM benefits from in-domain patterns, improving answer quality and relevance.

Figure 2: HyDE improving Dense and Term-based Retrieval.