[Paper Review] Overview of the TREC 2020 deep learning track

Nick Craswell, Bhaskar Mitra+2 more|arXiv (Cornell University)|Feb 15, 2021

Topic ModelingComputer Science9 references117 citations

TL;DR

The paper reports on the second TREC Deep Learning Track, comparing neural (especially BERT-style) versus traditional ranking methods in large training-data regimes for document and passage retrieval, and analyzes end-to-end versus rerank paradigms and the impact of the ORCAS dataset.

ABSTRACT

This is the second year of the TREC Deep Learning Track, with the goal of studying ad hoc ranking in the large training data regime. We again have a document retrieval task and a passage retrieval task, each with hundreds of thousands of human-labeled training queries. We evaluate using single-shot TREC-style evaluation, to give us a picture of which ranking methods work best when large data is available, with much more comprehensive relevance labeling on the small number of test queries. This year we have further evidence that rankers with BERT-style pretraining outperform other rankers in the large data regime.

Motivation & Objective

• Provide large, reusable training and test datasets for ad hoc ranking under a large-data regime.
• Establish a rigorous, blind single-shot evaluation to compare ranking methods.
• Evaluate document retrieval and passage retrieval with end-to-end and reranking setups.
• Analyze the impact of ORCAS data and different label schemes on evaluation.
• Encourage diverse approaches and analyze learnings over previous year.

Proposed method

• Provide two large training datasets (document and passage) with corresponding test collections.
• Include a large ORCAS click dataset as additional training data or as a document field.
• Classify submitted runs into nnlm, nn, and trad categories to compare pretrained-language-model–based, neural-network, and traditional methods.
• Evaluate runs using NDCG@10 with four-point NIST judgments, plus supplementary metrics (RR MS, RR, AP, NCg@k).
• Distinguish end-to-end fullrank versus reranking (rerank) to assess which phase contributes more to performance.
• Analyze cross-dataset agreement between MS MARCO and NIST labels, and compare the impact of ORCAS data on performance.

Experimental results

Research questions

• RQ1 Do BERT-style pretrained models (nnlm) outperform other methods in the large-data IR regime for document and passage retrieval?
• RQ2 What is the relative impact of end-to-end fullrank retrieval versus reranking on top-k ranking quality?
• RQ3 How does the ORCAS dataset affect performance and agreement between MS MARCO and NIST label-based evaluations?
• RQ4 Can traditional IR methods compete with neural approaches under extensive training data, and under what conditions do they fail?
• RQ5 How does evaluation label choice (NIST vs. MS MARCO) influence conclusions about model effectiveness?

Key findings

• nnlm runs outperform trad and nn runs on both tasks in terms of NDCG@10 across submitted results.
• For document retrieval, the best nnlm/run outperforms the best trad by 23% in NDCG@10 and the best nn by 11%; for passage retrieval, gaps are 42% and 17% respectively.
• End-to-end fullrank may not consistently surpass rerank in this track, with some top runs being rerank while fullrank occasionally matches or narrowly exceeds rerank.
• Using ORCAS data generally improves NDCG@10 within the same group, though it was not necessary to achieve the top score; the maximum observed improvement was about 0.0513 in NDCG@10.
• MS MARCO vs. NIST label agreement varies by task, with Kendall tau of 0.46 for document retrieval and 0.69 for passage retrieval; ORCAS usage influences these correlations.
• The track produced more diverse submissions (including more nnlm and fullrank/rerank comparisons) and provided reusable test collections for future research.