[Paper Review] BERT-QE: Contextualized Query Expansion for Document Re-ranking

Zhi Zheng, Kai Hui+4 more|arXiv (Cornell University)|Sep 15, 2020

Topic Modeling32 references4 citations

TL;DR

BERT-QE proposes a contextualized query expansion method that leverages BERT to identify and select relevant text chunks from top-ranked feedback documents for query expansion, significantly outperforming BERT-Large on TREC Robust04 and GOV2 benchmarks with only a 3-30% increase in computational cost.

ABSTRACT

Query expansion aims to mitigate the mismatch between the language used in a query and in a document. However, query expansion methods can suffer from introducing non-relevant information when expanding the query. To bridge this gap, inspired by recent advances in applying contextualized models like BERT to the document retrieval task, this paper proposes a novel query expansion model that leverages the strength of the BERT model to select relevant document chunks for expansion. In evaluation on the standard TREC Robust04 and GOV2 test collections, the proposed BERT-QE model significantly outperforms BERT-Large models.

Motivation & Objective

• Address the vocabulary mismatch problem in information retrieval by improving query expansion using contextualized representations.
• Overcome limitations of traditional query expansion methods that introduce non-relevant terms due to reliance on weak relevance signals or scalar statistics.
• Leverage BERT’s ability to identify highly relevant document chunks at a fine-grained level to enhance the quality of expansion terms.
• Achieve superior re-ranking performance compared to BERT-Large while maintaining computational efficiency through model variant trade-offs.
• Demonstrate the effectiveness of BERT-based chunk-level relevance scoring for query expansion in both shallow and deep retrieval pools.

Proposed method

• Fine-tune a BERT model on the target retrieval dataset (e.g., Robust04 or GOV2) using MS MARCO pretraining as initialization.
• In phase one, re-rank the initial document list using the fine-tuned BERT model to obtain top-k feedback documents.
• In phase two, split the top-ranked feedback documents into fixed-length text chunks (e.g., 10 tokens), and use the BERT model to score each chunk’s relevance to the original query.
• Select the top-k_c most relevant chunks based on BERT scores for expansion.
• In phase three, re-rank all documents using a combined query formed by the original query and the selected relevant chunks, with BERT scoring the final relevance.
• Use multiple BERT variants (e.g., BERT-Large, BERT-Base, BERT-Tiny) to explore efficiency-effectiveness trade-offs, resulting in models like BERT-QE-LLL, BERT-QE-LMT, and BERT-QE-LLS.

Experimental results

Research questions

• RQ1 Can BERT-based chunk-level relevance scoring improve query expansion quality compared to traditional token- or document-level methods?
• RQ2 How does the performance of BERT-QE compare to BERT-Large in standard TREC re-ranking benchmarks?
• RQ3 What is the optimal balance between computational cost and retrieval effectiveness when using smaller BERT variants in the BERT-QE framework?
• RQ4 Does selecting fine-grained, contextually relevant text chunks lead to better handling of vocabulary mismatch than standard query expansion techniques?
• RQ5 Can the proposed method achieve state-of-the-art performance with only a modest increase in inference cost?

Key findings

• BERT-QE-LLL, which uses BERT-Large in all three phases, significantly outperforms BERT-Large on both the TREC Robust04 and GOV2 test collections.
• BERT-QE-LMT, using a smaller BERT variant, achieves performance comparable to BERT-Large on the shallow pool with only a 3% increase in computational cost.
• BERT-QE-LLS, a larger variant, outperforms BERT-Large on both shallow and deep pools with a 30% increase in computation.
• The optimal chunk size for expansion is k_c = 10, as it balances performance and computational cost, with k_c = 20 showing only marginal gains.
• The model’s effectiveness is robust across different configurations, with m = 10 being the best setting for chunk selection.
• The use of BERT for fine-grained chunk relevance scoring enables more accurate and relevant query expansion than traditional methods relying on term frequency or language models.