[Paper Review] Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval

Zhuyun Dai, Jamie Callan|arXiv (Cornell University)|Oct 23, 2019

Topic ModelingComputer Science33 references117 citations

TL;DR

The paper introduces DeepCT, a framework that uses BERT-based contextualized representations to assign context-aware term weights for sentences/passages, enabling effective first-stage retrieval with offline indexing (DeepCT-Index) and query weighting (DeepCT-Query).

ABSTRACT

Term frequency is a common method for identifying the importance of a term in a query or document. But it is a weak signal, especially when the frequency distribution is flat, such as in long queries or short documents where the text is of sentence/passage-length. This paper proposes a Deep Contextualized Term Weighting framework that learns to map BERT's contextualized text representations to context-aware term weights for sentences and passages. When applied to passages, DeepCT-Index produces term weights that can be stored in an ordinary inverted index for passage retrieval. When applied to query text, DeepCT-Query generates a weighted bag-of-words query. Both types of term weight can be used directly by typical first-stage retrieval algorithms. This is novel because most deep neural network based ranking models have higher computational costs, and thus are restricted to later-stage rankers. Experiments on four datasets demonstrate that DeepCT's deep contextualized text understanding greatly improves the accuracy of first-stage retrieval algorithms.

Motivation & Objective

• Motivate the limitation of frequency-based term weighting in sentence- and passage-length text.
• Propose a deep contextualized term weighting framework that maps BERT embeddings to term weights.
• Show how to use context-aware weights for offline passage indexing (DeepCT-Index) and for weighting long queries (DeepCT-Query).
• Demonstrate improvements in first-stage retrieval accuracy on MS MARCO and TREC-CAR datasets.

Proposed method

• Use BERT to generate contextualized token embeddings for text passages and queries.
• Learn a linear regression layer on top of BERT embeddings to map contextualized features to non-negative term weights via mean-squared error loss.
• For passages, train on query-term recall targets to create TF-DeepCT weights and index offline in a standard inverted index (DeepCT-Index).
• For queries, train on term recall from relevant documents to create weighted bag-of-words or weighted SDM queries (DeepCT-Query).
• Compare against tf, TextRank, and Doc2Query baselines using BM25 and QL first-stage retrieval, with re-ranking by Conv-KNRM and BERT re-ranker.

Experimental results

Research questions

• RQ1 Can context-aware term weighting from contextualized representations improve first-stage retrieval for passages and long queries?
• RQ2 How does DeepCT-Index perform offline indexing and retrieval with standard bag-of-words models like BM25 and QL?
• RQ3 Does DeepCT-Query effectively re-weight long queries to improve initial retrieval?
• RQ4 What is the impact of using BERT versus non-contextual embeddings (W/E) on term weighting performance?
• RQ5 How does DeepCT-Index influence downstream re-ranking efficiency and effectiveness?

Key findings

• DeepCT-Index yields significant gains over tf, TextRank, and Doc2Query for BM25 and QL on MS MARCO and TREC-CAR.
• Contextualized embeddings (BERT) outperform ELMo and non-contextual word2vec in producing effective passage term weights.
• First-stage DeepCT-Index BM25 can surpass some multi-stage and neural re-ranking baselines while maintaining efficiency.
• Using DeepCT-Query to weight long queries improves retrieval when integrated with BM25 or QL, via weighted BOW or SDM formulations.
• First-stage improvements translate into better recall for re-rankers and can reduce the necessary depth of subsequent re-ranking.