[Paper Review] Multi-Vector Retrieval as Sparse Alignment

Yujie Qian, Jinhyuk Lee+6 more|arXiv (Cornell University)|Nov 2, 2022

Topic Modeling4 citations

TL;DR

ALIGNER is a novel multi-vector retrieval model that formulates document retrieval as sparse token-level alignment between query and document tokens, using learnable pairwise alignments and entropy-regularized unary saliences to enable task-adaptive sparsity and efficient pruning. It achieves state-of-the-art zero-shot performance (51.1 nDCG@10 on BEIR) and up to 15.7-point gains with few-shot adaptation, while reducing document token storage by 80% with minimal performance loss.

ABSTRACT

Multi-vector retrieval models improve over single-vector dual encoders on many information retrieval tasks. In this paper, we cast the multi-vector retrieval problem as sparse alignment between query and document tokens. We propose AligneR, a novel multi-vector retrieval model that learns sparsified pairwise alignments between query and document tokens (e.g. `dog' vs. `puppy') and per-token unary saliences reflecting their relative importance for retrieval. We show that controlling the sparsity of pairwise token alignments often brings significant performance gains. While most factoid questions focusing on a specific part of a document require a smaller number of alignments, others requiring a broader understanding of a document favor a larger number of alignments. Unary saliences, on the other hand, decide whether a token ever needs to be aligned with others for retrieval (e.g. `kind' from `kind of currency is used in new zealand}'). With sparsified unary saliences, we are able to prune a large number of query and document token vectors and improve the efficiency of multi-vector retrieval. We learn the sparse unary saliences with entropy-regularized linear programming, which outperforms other methods to achieve sparsity. In a zero-shot setting, AligneR scores 51.1 points nDCG@10, achieving a new retriever-only state-of-the-art on 13 tasks in the BEIR benchmark. In addition, adapting pairwise alignments with a few examples (<= 8) further improves the performance up to 15.7 points nDCG@10 for argument retrieval tasks. The unary saliences of AligneR helps us to keep only 20% of the document token representations with minimal performance loss. We further show that our model often produces interpretable alignments and significantly improves its performance when initialized from larger language models.

Motivation & Objective

• To address the inefficiency and suboptimal performance of existing multi-vector retrieval models, which rely on dense, non-sparse alignments and lack efficient pruning mechanisms.
• To enable task-specific sparsity in token-level alignments—favoring fewer alignments for factoid questions and more for broad-coverage tasks like argument retrieval.
• To develop an end-to-end differentiable method for learning sparse unary saliences that identify which query and document tokens are relevant for retrieval, enabling efficient pruning.
• To improve retrieval efficiency without sacrificing performance by reducing storage and computation costs through salience-based pruning of token representations.
• To demonstrate that sparse alignments are interpretable and that initialization from large language models further boosts performance.

Proposed method

• Formulates multi-vector retrieval as sparse alignment between query and document tokens, where each alignment represents a pairwise match between contextualized token vectors.
• Introduces pairwise alignments that are sparsely learned via a differentiable mechanism, allowing flexible control over the number of alignments per query token.
• Proposes unary saliences to determine whether a token should be aligned at all, effectively masking entire rows/columns in the alignment matrix.
• Employs entropy-regularized linear programming to learn sparse unary saliences end-to-end, outperforming L1 or gating mechanisms in sparsity and performance.
• Enables zero-shot inference with strong performance and few-shot adaptation by fine-tuning pairwise alignments with ≤8 examples per task.
• Uses salience scores to prune up to 80% of document token vectors and 50% of query token vectors with minimal nDCG@10 drop.

Experimental results

Research questions

• RQ1 Can sparse, task-adaptive token-level alignments improve retrieval performance across diverse IR tasks?
• RQ2 Can learnable unary saliences effectively identify and prune irrelevant query and document tokens without performance degradation?
• RQ3 Does controlling the sparsity of pairwise alignments lead to better generalization across different retrieval intents (e.g., factoid vs. argument retrieval)?
• RQ4 Can ALIGNER achieve state-of-the-art performance without contrastive pre-training or distillation?
• RQ5 To what extent do sparse alignments improve interpretability and compatibility with large language models?

Key findings

• ALIGNER achieves a new SOTA of 51.1 nDCG@10 on the BEIR benchmark in a zero-shot setting, outperforming prior retriever-only models without contrastive pre-training or distillation.
• With only 1–8 fine-tuning examples, ALIGNER improves argument retrieval performance by up to 15.7 points nDCG@10, demonstrating strong few-shot adaptability.
• Pruning with unary saliences reduces document token storage by 80% and query token usage by 50%, with less than a 1-point drop in nDCG@10.
• The model produces interpretable alignments, with high-salience tokens often matching semantically relevant terms (e.g., 'heart' aligns with 'heart' in the document).
• Initializing ALIGNER from larger language models significantly improves performance, indicating strong compatibility with pre-trained LM priors.
• The entropy-regularized linear programming method for salience learning outperforms L1 regularization and ReLU gating in both sparsity and retrieval accuracy.