[Paper Review] Condenser: a Pre-training Architecture for Dense Retrieval

Luyu Gao, Jamie Callan|arXiv (Cornell University)|Apr 16, 2021

Topic Modeling39 references8 citations

TL;DR

This paper proposes Condenser, a novel pre-training architecture that enhances Transformer language models for dense retrieval by explicitly conditioning language modeling objectives on a single dense representation, thereby improving structural readiness for bi-encoder fine-tuning. Condenser achieves state-of-the-art performance on multiple retrieval and similarity tasks, especially under low-data settings, outperforming standard BERT and matching task-specific models without complex training techniques.

ABSTRACT

Pre-trained Transformer language models (LM) have become go-to text representation encoders. Prior research fine-tunes deep LMs to encode text sequences such as sentences and passages into single dense vector representations for efficient text comparison and retrieval. However, dense encoders require a lot of data and sophisticated techniques to effectively train and suffer in low data situations. This paper finds a key reason is that standard LMs' internal attention structure is not ready-to-use for dense encoders, which needs to aggregate text information into the dense representation. We propose to pre-train towards dense encoder with a novel Transformer architecture, Condenser, where LM prediction CONditions on DENSE Representation. Our experiments show Condenser improves over standard LM by large margins on various text retrieval and similarity tasks.

Motivation & Objective

• To address the challenge of training effective bi-encoders from standard pre-trained language models, which suffer from poor structural readiness for dense representation aggregation.
• To investigate whether pre-training can be designed to inherently support bi-encoder fine-tuning, reducing the need for complex downstream training strategies.
• To improve performance in low-data scenarios where standard fine-tuned models often fail or underperform.
• To analyze the attention mechanisms of pre-trained models and demonstrate that structural readiness significantly affects training efficiency and final performance.
• To provide a general-purpose, lightweight alternative to task-specific pre-training methods for dense retrieval.

Proposed method

• Propose a new pre-training objective where language modeling is conditioned on a single dense representation, explicitly training the model to aggregate information into a compact vector.
• Modify the standard Transformer encoder architecture to include a dedicated dense representation token that attends to and aggregates information from the input sequence.
• Train the model using a masked language modeling objective, where the prediction at each token depends on both the input sequence and the dense representation, encouraging the model to learn to compress information effectively.
• Use the same architecture for both pre-training and fine-tuning, enabling direct transfer to downstream bi-encoder tasks such as sentence similarity and dense retrieval.
• Integrate hard negative mining during fine-tuning, demonstrating that Condenser is robust to noisy mined negatives, unlike standard BERT.
• Conduct ablation studies and attention analysis to compare the internal attention patterns of Condenser, standard BERT, and task-specific pre-trained models (e.g., ICT).

Experimental results

Research questions

• RQ1 Does standard pre-trained BERT have an internal attention structure that is inherently unsuitable for dense representation learning in bi-encoder setups?
• RQ2 Can pre-training be restructured to improve structural readiness for bi-encoder fine-tuning, reducing the need for complex downstream training techniques?
• RQ3 How does Condenser perform compared to standard BERT and task-specific pre-trained models under low-data and high-data settings?
• RQ4 To what extent does the attention mechanism of Condenser differ from that of standard BERT and ICT, and does this difference explain its improved training efficiency?
• RQ5 Can Condenser be effectively combined with hard negative mining, and does it maintain robustness to noisy mined negatives?

Key findings

• Condenser achieves significant performance gains over standard BERT on sentence similarity, question answering retrieval, and web search retrieval tasks, especially under low-data conditions.
• In low-data setups, Condenser matches or exceeds the performance of task-specific pre-trained models like those fine-tuned with Inverse Cloze Task (ICT), demonstrating its effectiveness as a general-purpose initializer.
• With sufficient training data, Condenser retrievers optimize more easily and outperform previous models that required complex techniques such as multi-round hard negative mining.
• Condenser is robust to noisy hard negatives, showing consistent performance improvements when trained with mined negatives, unlike BERT-based models whose performance degrades significantly under the same conditions.
• Attention analysis reveals that Condenser maintains a stable, broad attention pattern across layers, indicating structural readiness, while standard BERT undergoes drastic internal attention changes during fine-tuning, confirming its lack of structural readiness.
• The results suggest that task-specific pre-training objectives like ICT are not strictly necessary for structural readiness, as Condenser achieves similar benefits through a general pre-training objective focused on dense representation conditioning.