[Paper Review] Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval

Lee Xiong, Chenyan Xiong+6 more|arXiv (Cornell University)|Jul 1, 2020

Domain Adaptation and Few-Shot Learning61 references229 citations

TL;DR

The paper proposes ANCE, a global negative sampling method for dense text retrieval that uses an asynchronously updated ANN index to select hard negatives from the entire corpus, improving training convergence and retrieval accuracy. ANCE achieves state-of-the-art results across web search, OpenQA, and production search, with substantial efficiency gains.

ABSTRACT

Conducting text retrieval in a dense learned representation space has many intriguing advantages over sparse retrieval. Yet the effectiveness of dense retrieval (DR) often requires combination with sparse retrieval. In this paper, we identify that the main bottleneck is in the training mechanisms, where the negative instances used in training are not representative of the irrelevant documents in testing. This paper presents Approximate nearest neighbor Negative Contrastive Estimation (ANCE), a training mechanism that constructs negatives from an Approximate Nearest Neighbor (ANN) index of the corpus, which is parallelly updated with the learning process to select more realistic negative training instances. This fundamentally resolves the discrepancy between the data distribution used in the training and testing of DR. In our experiments, ANCE boosts the BERT-Siamese DR model to outperform all competitive dense and sparse retrieval baselines. It nearly matches the accuracy of sparse-retrieval-and-BERT-reranking using dot-product in the ANCE-learned representation space and provides almost 100x speed-up.

Motivation & Objective

• Identify the bottlenecks in learning for dense text retrieval with in-batch negatives.
• Propose a negative sampling strategy that uses global negatives from the entire corpus to improve learning convergence.
• Theoretically analyze gradient norms and convergence under negative sampling.
• Empirically validate ANCE across web search, OpenQA, and production search settings.
• Demonstrate efficiency gains of dense retrieval with ANCE over traditional approaches.

Proposed method

• The authors analyze the convergence of dense retrieval training and show local in-batch negatives yield diminishing gradient norms and high gradient variance.
• They introduce Approximate Nearest Neighbor Negative Contrastive Estimation (ANCE) to sample hard negatives globally from the corpus using an asynchronously updated ANN index.
• ANCE maintains an Inferencer that re-encodes the corpus checkpoint periodically and refreshes the ANN index to provide up-to-date negatives for training.
• Negatives for a training instance are drawn from the top retrieved documents (excluding positives) according to the current model, approximating an oracle importance sampling distribution.
• An asynchronous index refresh strategy balances training efficiency with using up-to-date negatives, typically refreshing every fixed number of batches.
• The implementation uses a BERT-Siamese/Dual Encoder with dot-product similarity and Negative Log Likelihood loss, trained with BM25-warmed initialization and standard pooling for long documents.

Experimental results

Research questions

• RQ1 Can global negative sampling from the full corpus improve dense retrieval learning over in-batch negatives?
• RQ2 How does asynchronous updating of the ANN index affect training efficiency and convergence?
• RQ3 What is the impact of ANCE on retrieval accuracy in web search, OpenQA, and production search systems?
• RQ4 How do ANCE negatives influence gradient norms and training dynamics compared to local negatives?

Key findings

• ANCE substantially improves dense retrieval performance over baselines using random or in-batch negatives.
• ANCE retrieves with dot-product similarity that nearly matches the accuracy of a BERT-based cascade IR pipeline while being far more efficient.
• Experiments in web search, OpenQA, and a commercial search engine show consistent gains from ANCE, including favorable propagation to reader-based questions answering stages.
• Theoretical and empirical analyses show ANCE negatives generate larger gradient norms and reduce gradient variance, accelerating convergence compared to local negatives.
• An asynchronous index refresh strategy keeps training efficient while maintaining up-to-date negatives, with reported practical GPU resource recommendations.
• Hardware-efficient dense retrieval via ANCE delivers about 100x faster processing than BERT reranking in online latency measurements in some setups.