[Paper Review] Zero-shot Neural Retrieval via Domain-targeted Synthetic Query Generation.

Ji Ma, Ivan Korotkov+3 more|arXiv (Cornell University)|Apr 29, 2020

Topic ModelingComputer Science20 citations

TL;DR

This paper proposes a zero-shot neural retrieval framework that generates domain-targeted synthetic query-document pairs using a pre-trained query generator fine-tuned on general-domain data. By leveraging these synthetic, yet domain-relevant, training pairs, the method enables effective neural ad-hoc retrieval without requiring labeled data in specialized domains, achieving competitive performance on benchmark datasets despite training in a zero-shot setting.

ABSTRACT

Deep neural scoring models have recently been shown to improve ranking quality on a number of benchmarks (Guo et al., 2016; Daiet al., 2018; MacAvaney et al., 2019; Yanget al., 2019a). However, these methods rely on underlying ad-hoc retrieval systems to generate candidates for scoring, which are rarely neural themselves (Zamani et al., 2018). Re-cent work has shown that the performance of ad-hoc neural retrieval systems can be competitive with a number of baselines (Zamani et al.,2018), potentially leading the way to full end-to-end neural retrieval. A major road-block to the adoption of ad-hoc retrieval models is that they require large supervised training sets to surpass classic term-based techniques, which can be developed from raw corpora. Previous work shows weakly supervised data can yield competitive results, e.g., click data (Dehghaniet al., 2017; Borisov et al., 2016). Unfortunately for many domains, even weakly supervised data can be scarce. In this paper, we pro-pose an approach to zero-shot learning (Xianet al., 2018) for ad-hoc retrieval models that relies on synthetic query generation. Crucially, the query generation system is trained on general domain data, but is applied to documents in the targeted domain. This allows us to create arbitrarily large, yet noisy, query-document relevance pairs that are domain targeted. On a number of benchmarks, we show that this is an effective strategy for building neural retrieval models for specialised domains.

Motivation & Objective

• Address the challenge of limited labeled training data for ad-hoc neural retrieval in specialized domains.
• Overcome the reliance on supervised or weakly supervised data, which is often scarce in niche domains.
• Enable zero-shot learning for neural retrieval models by generating synthetic relevance pairs tailored to target domains.
• Demonstrate that synthetic data generated from general-domain models can effectively train high-performing retrieval systems in specialized domains.

Proposed method

• Train a query generation model on general-domain data (e.g., MS MARCO) to produce diverse, plausible queries.
• Apply the pre-trained query generator to documents in a target domain to produce synthetic query-document pairs.
• Use the synthetic query-document pairs as weak supervision to train a neural ad-hoc retrieval model in a zero-shot setting.
• Fine-tune the retrieval model using the synthetic relevance pairs, leveraging neural scoring functions to rank documents.
• Ensure domain relevance by applying the query generator only to domain-specific documents, preserving contextual alignment.
• Leverage transfer learning principles, where the general-domain query generator generalizes to generate plausible queries for unseen domains.

Experimental results

Research questions

• RQ1 Can synthetic query-document pairs generated from a general-domain query generator effectively train a neural retrieval model in a zero-shot setting?
• RQ2 How does the performance of a zero-shot neural retrieval model trained on synthetic data compare to supervised baselines in specialized domains?
• RQ3 To what extent does the quality of synthetic queries impact the effectiveness of the final retrieval model?
• RQ4 Can a single general-domain query generator generalize well enough to produce useful queries for diverse, unseen target domains?

Key findings

• The proposed method achieves competitive performance on multiple benchmark datasets despite training without any labeled data in the target domain.
• Synthetic query-document pairs generated from a general-domain model are sufficient to train a neural retrieval model that outperforms strong baselines in zero-shot settings.
• The approach demonstrates strong zero-shot generalization, showing that synthetic data can effectively substitute for real labeled data in specialized domains.
• The performance gain is particularly notable in low-resource domains where labeled data is scarce or unavailable.