[Paper Review] Sparse, Dense, and Attentional Representations for Text Retrieval

Yi Luan, Jacob Eisenstein+2 more|arXiv (Cornell University)|May 1, 2020

Domain Adaptation and Few-Shot Learning54 references83 citations

TL;DR

The paper analyzes the capacity of dense dual encoders vs sparse bag-of-words models and attentional networks for text retrieval, introduces multi-vector encodings and sparse-dense hybrids, and shows these hybrids and multi-vector approaches achieve strong large-scale retrieval performance across benchmarks.

ABSTRACT

Dual encoders perform retrieval by encoding documents and queries into dense lowdimensional vectors, scoring each document by its inner product with the query. We investigate the capacity of this architecture relative to sparse bag-of-words models and attentional neural networks. Using both theoretical and empirical analysis, we establish connections between the encoding dimension, the margin between gold and lower-ranked documents, and the document length, suggesting limitations in the capacity of fixed-length encodings to support precise retrieval of long documents. Building on these insights, we propose a simple neural model that combines the efficiency of dual encoders with some of the expressiveness of more costly attentional architectures, and explore sparse-dense hybrids to capitalize on the precision of sparse retrieval. These models outperform strong alternatives in large-scale retrieval.

Motivation & Objective

• Assess the capacity and fidelity of compressive (dense) dual encoders relative to sparse bag-of-words models for retrieval.
• Investigate how document length and encoding dimension affect retrieval fidelity and margins between top results.
• Propose architectures that combine dense representations with sparse or multiple vectors to improve retrieval efficiency and accuracy.
• Evaluate models on open-domain QA and MS MARCO benchmarks to determine practical effectiveness for large-scale retrieval.

Proposed method

• Theoretical analysis of compressive dual encoders using random projections to relate embedding dimension to fidelity with sparse bag-of-words retrieval.
• Derivation of bounds on pairwise ranking error and recall-at-r using random Gaussian or Rademacher embeddings (Lemmas 1–3).
• Introduction of a multi-vector encoding model where a document is represented by a set of vectors and relevance is the max over inner products with the query vector.
• Analysis of a cross-attention extension and comparison to dense and sparse baselines.
• Empirical evaluation across tasks: containing passage ICT, Natural Questions (reranking and open-domain retrieval), and MS MARCO, with BM25, DE-BERT variants, ME-BERT variants, and sparse-dense hybrids.
• Use of scalable nearest-neighbor search (ScaNN) for retrieval in large collections; training with a cross-entropy loss and hard-negative mining.

Experimental results

Research questions

• RQ1 What is the fidelity of compressed dense encodings relative to sparse bag-of-words models across document lengths?
• RQ2 How does document length and embedding size k affect the margin between gold and competing documents in dual-encoder setups?
• RQ3 Can a multi-vector or sparse-dense hybrid approach achieve higher retrieval accuracy and efficiency than traditional dual encoders or pure sparse methods, especially for long documents?
• RQ4 How do dense and hybrid models perform on large-scale retrieval benchmarks such as MS MARCO and Natural Questions, compared to BM25 and cross-attention reranking models?

Key findings

• Random projection theory shows embedding size k required for a given error probability scales with the normalized margin and document length.
• Multi-vector encodings (ME-BERT) outperform single-vector dual encoders (DE-BERT) and BM25 in several long-document retrieval settings.
• Cross-attentional models yield strong reranking performance but are computationally costly for large-scale retrieval; multi-vector and hybrids offer favorable efficiency-accuracy trade-offs.
• Sparse-dense hybrids (e.g., HYBRID-ME-BERT-uni/bi) provide notable gains over their components, particularly as document length increases.
• On MS MARCO and Natural Questions benchmarks, the hybrid and multi-vector approaches are competitive or superior to state-of-the-art retrieval methods, with ME-BERT-768 and related hybrids performing well across tasks.