[Paper Review] In Defense of Cross-Encoders for Zero-Shot Retrieval

Guilherme J. M. Rosa, Luiz Bonifacio+5 more|arXiv (Cornell University)|Dec 12, 2022

Domain Adaptation and Few-Shot Learning6 citations

TL;DR

This paper defends cross-encoders as superior to bi-encoders for zero-shot retrieval, demonstrating that larger cross-encoders with early query-document interaction and higher parameter counts generalize significantly better across out-of-domain tasks. The study shows that cross-encoders surpass state-of-the-art bi-encoders by over 4 average points on the BEIR benchmark, and that first-stage dense retrieval offers no advantage over BM25 when followed by cross-encoder reranking.

ABSTRACT

Bi-encoders and cross-encoders are widely used in many state-of-the-art retrieval pipelines. In this work we study the generalization ability of these two types of architectures on a wide range of parameter count on both in-domain and out-of-domain scenarios. We find that the number of parameters and early query-document interactions of cross-encoders play a significant role in the generalization ability of retrieval models. Our experiments show that increasing model size results in marginal gains on in-domain test sets, but much larger gains in new domains never seen during fine-tuning. Furthermore, we show that cross-encoders largely outperform bi-encoders of similar size in several tasks. In the BEIR benchmark, our largest cross-encoder surpasses a state-of-the-art bi-encoder by more than 4 average points. Finally, we show that using bi-encoders as first-stage retrievers provides no gains in comparison to a simpler retriever such as BM25 on out-of-domain tasks. The code is available at https://github.com/guilhermemr04/scaling-zero-shot-retrieval.git

Motivation & Objective

• To evaluate the zero-shot generalization performance of cross-encoders versus bi-encoders across diverse model sizes and domains.
• To investigate whether increasing model size improves generalization in out-of-domain retrieval scenarios.
• To assess whether bi-encoders as first-stage retrievers provide any advantage over simpler methods like BM25 when followed by cross-encoder reranking.
• To determine the role of early query-document interaction and hidden dimension scaling in cross-encoder generalization.

Proposed method

• Training and evaluating cross-encoders and bi-encoders across a wide range of model sizes, from 60M to 5.8B parameters, on the BEIR benchmark.
• Using monoT5-based cross-encoders as rerankers and comparing them to bi-encoders like GTR and SGPT across 22 retrieval tasks.
• Evaluating zero-shot effectiveness on out-of-domain datasets without fine-tuning, including TREC-COVID, Natural Questions, and FEVER.
• Comparing first-stage retrieval using BM25 versus GTR-335M, followed by monoT5-220M reranking, to assess the impact of initial retrieval quality.
• Analyzing the effect of hidden dimension scaling (e.g., 768 to 4096) on model generalization as parameter count increases.
• Measuring performance using nDCG@10 and MRR@10 across diverse domains, excluding MS MARCO where MRR@10 is used.

Experimental results

Research questions

• RQ1 How does model size affect the zero-shot generalization of cross-encoders versus bi-encoders across diverse domains?
• RQ2 Does early query-document interaction in cross-encoders contribute to better out-of-domain performance compared to bi-encoders?
• RQ3 Can dense retrievers (bi-encoders) outperform sparse methods like BM25 in multi-stage retrieval pipelines when followed by cross-encoder reranking?
• RQ4 Does increasing model size lead to marginal gains in-domain but substantial gains in out-of-domain generalization for cross-encoders?
• RQ5 To what extent does hidden dimension scaling with model size improve zero-shot effectiveness in cross-encoders?

Key findings

• Increasing model size leads to marginal gains in-domain but substantial improvements in out-of-domain generalization for cross-encoders.
• The largest cross-encoder (5.8B parameters) surpasses a state-of-the-art bi-encoder by more than 4 average points on the BEIR benchmark.
• Cross-encoders with early query-document interaction and higher hidden dimensions (e.g., 4096) show significantly better zero-shot performance than those with fixed 768 dimensions.
• Using bi-encoders like GTR as first-stage retrievers provides no performance advantage over BM25 when followed by cross-encoder reranking.
• The effectiveness of cross-encoders remains high regardless of the initial retrieval method, indicating that reranking quality is the dominant factor in multi-stage pipelines.
• Cross-encoders outperform similarly sized bi-encoders across all tested domains, including legal, medical, and scientific text, confirming their superior generalization capability.