[Paper Review] Complementing Lexical Retrieval with Semantic Residual Embedding

Luyu Gao, Zhuyun Dai+4 more|arXiv (Cornell University)|Apr 29, 2020

Topic ModelingComputer Science45 references59 citations

TL;DR

CLEAR combines BM25-like lexical retrieval with a neural embedding model trained to complement lexical signals through a residual learning scheme, improving first-stage retrieval and downstream reranking efficiency.

ABSTRACT

This paper presents CLEAR, a retrieval model that seeks to complement classical lexical exact-match models such as BM25 with semantic matching signals from a neural embedding matching model. CLEAR explicitly trains the neural embedding to encode language structures and semantics that lexical retrieval fails to capture with a novel residual-based embedding learning method. Empirical evaluations demonstrate the advantages of CLEAR over state-of-the-art retrieval models, and that it can substantially improve the end-to-end accuracy and efficiency of reranking pipelines.

Motivation & Objective

• Motivate improving first-stage retrieval by combining exact lexical matching with semantic embedding signals.
• Develop a residual-based training method for embedding models to complement lexical retrieval and fix lexical errors.
• Demonstrate that CLEAR outperforms traditional lexical baselines and embedding-only models on large-scale datasets.
• Show that CLEAR provides additive gains to downstream BERT rerankers and reduces end-to-end retrieval costs.

Proposed method

• Implement a dual-branch retrieval system: a BM25-style lexical retriever and a Transformer-based embedding retriever (Siamese BERT with average-pooled embeddings).
• Train embeddings with a residual loss that augments lexical signals rather than replacing them, using a triplet hinge loss with an error-based negative sampling strategy.
• Introduce a residual margin m_r that depends on lexical scores to focus embedding learning on lexical weaknesses.
• Use a union of lexical and embedding candidate lists and an interpolated final score s_CLEAR = lambda_test * s_lex + s_emb.
• Leverage fast MIPS indexing for embedding retrieval and inverted indexes for lexical retrieval to enable scalable, single-stage retrieval.

Experimental results

Research questions

• RQ1 Can a residual-based embedding training objective improve complementarity between lexical and neural retrieval models?
• RQ2 Does combining lexical and embedding retrieval in CLEAR yield higher first-stage retrieval metrics than lexical or embedding models alone?
• RQ3 How does CLEAR affect end-to-end performance when paired with BERT rerankers?
• RQ4 What are the qualitative trade-offs (false positives, reranker behavior) when using semantic residual embeddings in first-stage retrieval?

Key findings

• CLEAR achieves state-of-the-art first-stage retrieval effectiveness on MS MARCO and outperforms both lexical baselines and embedding-only models.
• Embedding models trained with residual margins and error-based negative sampling better complement lexical signals than post-hoc fusion approaches.
• In pipelines with BERT rerankers, CLEAR reduces the required reranking depth and improves end-to-end accuracy and efficiency.
• BERT rerankers can still struggle with semantically related false positives introduced by CLEAR, highlighting new challenges for neural rerankers.