[Paper Review] Understanding the Behaviors of BERT in Ranking

Yifan Qiao, Chenyan Xiong+2 more|arXiv (Cornell University)|Apr 16, 2019

Topic Modeling8 references146 citations

TL;DR

The paper analyzes how BERT-based rankers perform on MS MARCO passage ranking and TREC Web Track ad hoc ranking, showing strong gains on MS MARCO due to cross-query interactions, but weaker gains on TREC-style ranking; it also investigates BERT's attention patterns and term matching behaviors.

ABSTRACT

This paper studies the performances and behaviors of BERT in ranking tasks. We explore several different ways to leverage the pre-trained BERT and fine-tune it on two ranking tasks: MS MARCO passage reranking and TREC Web Track ad hoc document ranking. Experimental results on MS MARCO demonstrate the strong effectiveness of BERT in question-answering focused passage ranking tasks, as well as the fact that BERT is a strong interaction-based seq2seq matching model. Experimental results on TREC show the gaps between the BERT pre-trained on surrounding contexts and the needs of ad hoc document ranking. Analyses illustrate how BERT allocates its attentions between query-document tokens in its Transformer layers, how it prefers semantic matches between paraphrase tokens, and how that differs with the soft match patterns learned by a click-trained neural ranker.

Motivation & Objective

• Investigate how pre-trained BERT can be leveraged for ranking tasks and its effectiveness across benchmarks.
• Compare representation-based vs interaction-based BERT ranking setups.
• Analyze BERT's attention distributions and term-matching behavior during ranking.
• Explain why BERT excels on MS MARCO but not as strongly on TREC ad hoc ranking.

Proposed method

• Fine-tune four BERT-based ranking models (BERT Rep, Last-Int, Mult-Int, Term-Trans) on MS MARCO and ClueWeb data.
• Use pre-trained BERT-Large as the backbone with end-to-end fine-tuning for relevance classification.
• Represent q, d, or qd sequences using CLS embeddings and linear or cross-attention-based scoring.
• Evaluate against BM25, LeToR, K-NRM, Conv-KNRM baselines on MS MARCO and ClueWeb with standard IR metrics.
• Analyze attention distributions and term-matching behavior to understand what BERT attends to and how it matches terms.

Experimental results

Research questions

• RQ1 How do different BERT-based ranking formulations (Rep, Last-Int, Mult-Int, Term-Trans) perform on MS MARCO passage ranking and on TREC Web Track ad hoc ranking?
• RQ2 To what extent do cross-query-document interactions in BERT drive ranking performance?
• RQ3 How do BERT’s attention patterns and influential terms differ from traditional neural rankers and from user-click based pre-training signals?
• RQ4 Why does BERT excel in MS MARCO but not in TREC ad hoc tasks, and what signals are more effective for each setting?

Key findings

• BERT-based rankers substantially improve MS MARCO performance, with interaction-based variants (Last-Int, Mult-Int, Term-Trans) outperforming earlier models and strong cross-question-passage interactions driving gains.
• BERT (Rep), which treats q and d separately, performs near random on MS MARCO, highlighting the importance of cross-sequence interactions for ranking.
• On ClueWeb/TREC ad hoc tasks, BERT-based rankers do not outperform feature-based LeToR or a Bing-click pre-trained model (Conv-KNRM Bing); MARCO pre-training signals do not transfer as well to ad hoc ranking.
• BERT attends more to markers ([CLS], [SEP]) and less to stopwords, with attention becoming more distributed deeper in the network; markers are crucial for distinguishing sequences.
• Influential terms in BERT (Last-Int) are often exact-match or close paraphrases from the query, while Conv-KNRM relies on looser semantic associations.
• BERT’s pre-training on surrounding contexts favors semantically close text pairs, aligning with seq2seq-style ranking, whereas ad hoc ranking benefits more from user-click pre-training signals.