[Paper Review] Models and Data for Simple Applications of BERT for Ad Hoc Document Retrieval

Yang, Wei, Wei Yang+1 more|arXiv (Cornell University)|Mar 26, 2019

Topic ModelingComputer Science23 references133 citations

TL;DR

The paper demonstrates a simple sentence-level BERT approach for ad hoc document retrieval, aggregating sentence scores to rank longer documents, achieving strong results on Microblog and Robust04 datasets.

ABSTRACT

Following recent successes in applying BERT to question answering, we explore simple applications to ad hoc document retrieval. This required confronting the challenge posed by documents that are typically longer than the length of input BERT was designed to handle. We address this issue by applying inference on sentences individually, and then aggregating sentence scores to produce document scores. Experiments on TREC microblog and newswire test collections show that our approach is simple yet effective, as we report the highest average precision on these datasets by neural approaches that we are aware of.

Motivation & Objective

• Motivate applying BERT to ad hoc document retrieval despite long documents and limited sentence-level relevance data.
• Propose a simple inference and aggregation technique that avoids complex fine-tuning on document-level labels.
• Evaluate the method on TREC Microblog Tracks and Robust04 to establish baseline neural performance.
• Show that sentence-level inference with score aggregation can yield competitive or superior results to prior neural models.

Proposed method

• Use Anserini for initial retrieval and BERT for sentence-level relevance classification.
• Fine-tune BERT on available sentence-level or related data (microblog, QA, WikiQA) and apply binary relevance classification with CLS embedding.
• For short documents (microblogs), concatenate query and document as input to BERT and interpolate BERT scores with IR scores.
• For longer documents (newswire), compute BERT scores for top sentences and aggregate with original document scores using a weighted sum with hyperparameters a and w_i.
• Tune interpolation weights and sentence counts (top n sentences) via cross-validation.
• Report AP and P30 as evaluation metrics and compare against BM25+RM3 and various neural baselines.

Experimental results

Research questions

• RQ1 Can BERT be effectively applied to ad hoc document retrieval given the length mismatch between documents and BERT input limits?
• RQ2 Does sentence-level inference with score aggregation yield competitive or superior performance to traditional neural ranking models on standard datasets?
• RQ3 What is the impact of fine-tuning data source (microblog vs QA/WikiQA) on the effectiveness of BERT for retrieval?
• RQ4 How does aggregating top-scoring sentences compare to using the full document score for retrieval?

Key findings

• BERT-based scoring with simple sentence-level inference improves over prior neural models on Microblog tracks, achieving substantial gains in AP and P30.
• On Robust04, fine-tuning BERT on microblog data outperforms QA-based fine-tuning, suggesting task relevance matters more than document genre.
• The best results for Robust04 come from using the top three sentences; adding a fourth sentence does not help under the tuned settings.
• BM25+RM3 remains a strong baseline and, in some settings, outperforms neural models, but the proposed BERT-based reranker yields further significant improvements.
• Across evaluated datasets, the simple sentence-level aggregation approach yields state-of-the-art results among neural methods reported at the time for these tasks.