[Paper Review] RepBERT: Contextualized Text Embeddings for First-Stage Retrieval

Jingtao Zhan, Jiaxin Mao+3 more|arXiv (Cornell University)|Jun 28, 2020

Topic ModelingComputer Science21 references58 citations

TL;DR

RepBERT uses fixed-length contextualized embeddings for queries and documents, scoring relevance by inner product, achieving state-of-the-art first-stage retrieval on MS MARCO while remaining as efficient as bag-of-words methods.

ABSTRACT

Although exact term match between queries and documents is the dominant method to perform first-stage retrieval, we propose a different approach, called RepBERT, to represent documents and queries with fixed-length contextualized embeddings. The inner products of query and document embeddings are regarded as relevance scores. On MS MARCO Passage Ranking task, RepBERT achieves state-of-the-art results among all initial retrieval techniques. And its efficiency is comparable to bag-of-words methods.

Motivation & Objective

• Motivate replacing bag-of-words first-stage retrieval with semantic, fixed-length embeddings.
• Develop an encoder-based model that produces comparable online efficiency to BOW methods.
• Demonstrate state-of-the-art first-stage retrieval performance on MS MARCO Passage Ranking.
• Analyze training strategies and combination with exact-match signals.
• Provide insights into reranking compatibility and practical deployment considerations.

Proposed method

• Use BERT-based encoder to produce fixed-length embeddings for queries and documents by averaging token representations from the input with [CLS]/[SEP] framing.
• Share encoder weights between query and document representations to enable semantic matching.
• Define relevance as the inner product of query and document embeddings and optimize with a MultiLabelMarginLoss across in-batch negatives.
• Employ in-batch negative sampling to efficiently train with many negative examples within a batch.
• Train with Train Triples data from MS MARCO and fine-tune using ADAM with specified hyperparameters (learning rate 3e-6, warmup, etc.).
• Evaluate first-stage retrieval performance against BM25 and other baselines on MS MARCO Passage Ranking.

Experimental results

Research questions

• RQ1 Can fixed-length contextualized embeddings for queries and documents replace traditional bag-of-words approaches for first-stage retrieval?
• RQ2 What is the impact of semantic matching via embeddings on retrieval metrics (MRR@10, Recall@1000) compared to BM25 and other neural methods?
• RQ3 How does RepBERT interact with reranking models and other retrieval signals (e.g., docTTTTTquery) in terms of recall and ranking performance?
• RQ4 What training strategies (in-batch negatives) and model configurations yield the best first-stage retrieval performance?

Key findings

• RepBERT achieves higher MRR@10 than BM25, doc2query, DeepCT, and docTTTTTquery in first-stage retrieval on MS MARCO dev/test.
• RepBERT delivers Recall@1000 close to the best DocTTTTTquery result, outperforming most baselines on large candidate sets.
• its efficiency is comparable to bag-of-words methods, with offline embedding storage and online inner-product computations.
• When used as a first-stage retriever, RepBERT provides the best recall at smaller depths and competitive performance at larger depths for subsequent reranking with BERT Large.
• Combining RepBERT with exact-match retrievers (e.g., docTTTTTquery, BM25) yields improved MRR@10 and Recall@1000, indicating complementary strengths.