[Paper Review] Promptagator: Few-shot Dense Retrieval From 8 Examples

Zhuyun Dai, Vincent Y. Zhao+8 more|arXiv (Cornell University)|Sep 23, 2022

Topic Modeling46 citations

TL;DR

Promptagator uses a few-shot prompt-based LLM query generator to create synthetic task-specific data, training small end-to-end dual-encoder retrievers that outperform MS MARCO-trained models on BEIR tasks, with an additional reranking boost.

ABSTRACT

Much recent research on information retrieval has focused on how to transfer from one task (typically with abundant supervised data) to various other tasks where supervision is limited, with the implicit assumption that it is possible to generalize from one task to all the rest. However, this overlooks the fact that there are many diverse and unique retrieval tasks, each targeting different search intents, queries, and search domains. In this paper, we suggest to work on Few-shot Dense Retrieval, a setting where each task comes with a short description and a few examples. To amplify the power of a few examples, we propose Prompt-base Query Generation for Retriever (Promptagator), which leverages large language models (LLM) as a few-shot query generator, and creates task-specific retrievers based on the generated data. Powered by LLM's generalization ability, Promptagator makes it possible to create task-specific end-to-end retrievers solely based on a few examples {without} using Natural Questions or MS MARCO to train %question generators or dual encoders. Surprisingly, LLM prompting with no more than 8 examples allows dual encoders to outperform heavily engineered models trained on MS MARCO like ColBERT v2 by more than 1.2 nDCG on average on 11 retrieval sets. Further training standard-size re-rankers using the same generated data yields another 5.0 point nDCG improvement. Our studies determine that query generation can be far more effective than previously observed, especially when a small amount of task-specific knowledge is given.

Motivation & Objective

• Highlight the diversity of retrieval tasks and the need for task-specific few-shot retrieval settings.
• Propose Promptagator to generate synthetic task-specific training data via LLM prompting without fine-tuning.
• Demonstrate that small dual encoders trained on generated data can outperform MS MARCO-trained models on BEIR tasks.
• Show that a subsequent reranker trained on the same data yields additional performance gains.

Proposed method

• Define a Few-shot Retrieval setting on BEIR with 2–8 in-domain examples per task.
• Use a large language model (FLAN 137B) to generate synthetic queries conditioned on task descriptions and few examples.
• Apply a round-trip consistency filter by training a preliminary retriever on synthetic data and keeping pairs where the source document ranks highly.
• Train a dual-encoder retriever initialized from a T5-based encoder on the synthetic data, followed by fine-tuning on filtered data.
• Train a cross-attention reranker (Promptagator++) on the same synthetic data to refine top candidates.
• Provide zero-shot and few-shot variants of Promptagator and compare to MS MARCO-trained baselines.

Experimental results

Research questions

• RQ1 Can a few-shot prompt-based data generation paradigm enable effective end-to-end dense retrievers without in-domain annotated query-document pairs?
• RQ2 How does prompt-based synthetic data quality, aided by consistency filtering, affect retrieval performance across diverse BEIR tasks?
• RQ3 What is the comparative impact of few-shot versus zero-shot prompt-generated data on retrieval and reranking performance?
• RQ4 How does Promptagator compare to MS MARCO-trained baselines and to specialized rerankers on BEIR?

Key findings

• Zero-shot Promptagator establishes a strong baseline that rivals baselines trained on MS MARCO data.
• Few-shot Promptagator significantly improves over zero-shot, increasing average nDCG@10 by over 2 points on BEIR datasets.
• Promptagator outperforms strong MS MARCO-trained models such as ColBERT v2 and SPLADE v2 on 11 BEIR tasks.
• Promptagator++ (a cross-attention reranker) adds about 5 points to nDCG@10 beyond Promptagator, surpassing several reranking approaches.
• Consistency filtering improves performance on the majority of datasets and demonstrates the value of synthetic data quality control.