[Paper Review] Learning Diverse Document Representations with Deep Query Interactions for Dense Retrieval

Zehan Li, Nan Yang+2 more|arXiv (Cornell University)|Aug 8, 2022

Advanced Image and Video Retrieval Techniques4 citations

TL;DR

This paper proposes a dual-cross-encoder model that learns diverse, query-informed document representations by generating multiple pseudo-queries per document and encoding each document with its interaction to these queries. The method achieves cross-encoder-level performance with dual-encoder inference efficiency, outperforming strong baselines on MS MARCO, TREC DL 19, and TREC DL 20 benchmarks.

ABSTRACT

In this paper, we propose a new dense retrieval model which learns diverse document representations with deep query interactions. Our model encodes each document with a set of generated pseudo-queries to get query-informed, multi-view document representations. It not only enjoys high inference efficiency like the vanilla dual-encoder models, but also enables deep query-document interactions in document encoding and provides multi-faceted representations to better match different queries. Experiments on several benchmarks demonstrate the effectiveness of the proposed method, out-performing strong dual encoder baselines.The code is available at \url{https://github.com/jordane95/dual-cross-encoder

Motivation & Objective

• To bridge the performance gap between cross-encoder and dual-encoder models in dense retrieval without sacrificing inference efficiency.
• To improve document representation diversity by incorporating deep query-document interactions during encoding.
• To enable multi-faceted document representations that better match varying query semantics.
• To explore the effectiveness of pseudo-query generation as a data augmentation and representation learning strategy.

Proposed method

• For each document, a query generation model produces multiple pseudo-queries to serve as interaction prompts.
• Each document is encoded via a cross-encoder architecture by concatenating it with a generated query, creating query-informed, multi-view representations.
• The final document representation is obtained by max-pooling across all query-specific representations, preserving diversity and enabling efficient vector search.
• The model uses contrastive learning with hard negatives to optimize the representation space, improving zero-shot generalization.
• Parameter sharing between query and document encoders reduces model size while maintaining performance.
• The approach is compatible with maximum inner product search (MIPS), enabling efficient inference on pre-computed document vectors.

Experimental results

Research questions

• RQ1 Can deep query-document interactions be pre-computed to achieve cross-encoder-level performance with dual-encoder inference efficiency?
• RQ2 How does query diversity and quality affect retrieval performance in multi-view document encoding?
• RQ3 Can pseudo-query generation effectively augment training data and improve representation quality?
• RQ4 Does multi-view document encoding improve matching accuracy across diverse queries?

Key findings

• The proposed model outperforms the DPR dual-encoder baseline by 2.8 MRR@10 on MS MARCO (36.0 vs. 34.2) and 5.6 nDCG@10 on TREC DL 20 (68.9 vs. 63.3).
• On TREC DL 20, the model surpasses ColBERT in nDCG@10 while maintaining lower computational cost.
• Even without data augmentation, the model achieves comparable performance to DPR with data augmentation on the TREC DL 19 benchmark.
• Ablation shows that pre-training on generated queries improves fine-tuning performance, though direct fine-tuning on real data yields better results.
• Tying encoder parameters slightly reduces performance, indicating that asymmetric encoding is beneficial for modeling query-document interactions.
• The model achieves strong performance with only 4–6 generated queries per document, balancing diversity and efficiency.