[Paper Review] Aggretriever: A Simple Approach to Aggregate Textual Representations for Robust Dense Passage Retrieval

Sheng-Chieh Lin, Minghan Li+1 more|arXiv (Cornell University)|Jul 31, 2022

Topic Modeling4 citations

TL;DR

Aggretriever proposes a simple yet effective method to enhance dense passage retrieval by aggregating contextualized token embeddings from BERT's masked language modeling (MLM) head into a compact vector, $\mathbf{agg}^{*}$, which is concatenated with the [CLS] vector. This fusion improves retrieval performance on both in-domain and zero-shot benchmarks without additional pre-training or expensive fine-tuning.

ABSTRACT

Pre-trained language models have been successful in many knowledge-intensive NLP tasks. However, recent work has shown that models such as BERT are not ``structurally ready'' to aggregate textual information into a [CLS] vector for dense passage retrieval (DPR). This ``lack of readiness'' results from the gap between language model pre-training and DPR fine-tuning. Previous solutions call for computationally expensive techniques such as hard negative mining, cross-encoder distillation, and further pre-training to learn a robust DPR model. In this work, we instead propose to fully exploit knowledge in a pre-trained language model for DPR by aggregating the contextualized token embeddings into a dense vector, which we call agg*. By concatenating vectors from the [CLS] token and agg*, our Aggretriever model substantially improves the effectiveness of dense retrieval models on both in-domain and zero-shot evaluations without introducing substantial training overhead. Code is available at https://github.com/castorini/dhr

Motivation & Objective

• Address the limitation of BERT-based dense retrievers that underperform on out-of-domain and zero-shot settings due to poor aggregation of contextualized representations.
• Overcome the 'structural unready' issue of BERT's [CLS] vector, which fails to effectively capture fine-grained textual information from pre-training.
• Develop a computationally lightweight method to fully exploit pre-trained knowledge in BERT for dense retrieval without further pre-training or complex fine-tuning.
• Improve retrieval effectiveness by combining semantic (via [CLS]) and lexical (via MLM projection) signals into a single dense vector representation.

Proposed method

• Re-use the pre-trained MLM head of BERT to project each contextualized token embedding into the high-dimensional wordpiece vocabulary space.
• Apply max-pooling and pruning to the projected vectors to obtain a compact, high-impact lexical representation, denoted as $\mathbf{agg}^{\star}$.
• Concatenate $\mathbf{agg}^{\star}$ with the [CLS] vector to form the final passage representation, enabling joint modeling of semantic and lexical information.
• Fine-tune the resulting Aggretriever model using standard bi-encoder training with standard negative sampling, avoiding hard negative mining or distillation.
• Leverage existing ANN libraries for efficient approximate nearest neighbor search using the final dense vector representation.
• Ensure compatibility with existing retrieval pipelines by maintaining a single dense vector output, unlike hybrid models requiring dual representations.

Experimental results

Research questions

• RQ1 Can we improve dense passage retrieval by fully exploiting the pre-trained MLM head of BERT without further pre-training?
• RQ2 Does aggregating contextualized token embeddings into a compact vector $\mathbf{agg}^{\star}$ lead to better retrieval performance than relying solely on the [CLS] vector?
• RQ3 Can the proposed aggregation method improve zero-shot and out-of-domain retrieval effectiveness without introducing computational overhead?
• RQ4 Is the Aggretriever approach compatible with existing retrieval systems and efficient for ANN-based retrieval?

Key findings

• Aggretriever achieves significant improvements over standard [CLS]-only models on both in-domain and zero-shot retrieval benchmarks, with consistent gains across datasets.
• The model outperforms strong baselines such as BERT$_{\text{CLS}}$ and TAS-B, even when those models use knowledge distillation or hard negative mining.
• The addition of $\mathbf{agg}^{\star}$ to the [CLS] vector improves retrieval effectiveness without requiring expensive training techniques like hard negative mining or cross-encoder distillation.
• Aggretriever maintains strong performance on zero-shot settings, demonstrating robustness to distributional shift compared to standard DPR models.
• The method is orthogonal to further pre-training strategies, and combining Aggretriever with models like coCondenser further boosts performance.
• The MLM-based aggregation provides a stable lexical signal that enhances matching even in sparse retrieval, suggesting dual benefits for semantic and lexical matching.