[Paper Review] Overview of the TREC 2022 deep learning track

Nick Craswell, Bhaskar Mitra+5 more|ArXiv.org|Jul 10, 2025

Topic Modeling32 references15 citations

TL;DR

The paper reports on the fourth TREC Deep Learning Track, focusing on passage ranking with a larger MS MARCO-based test collection, and infers document rankings from passage judgments; neural ranking with large-scale pretraining outperforms traditional methods, and the test collection emphasizes reusability.

ABSTRACT

This is the fourth year of the TREC Deep Learning track. As in previous years, we leverage the MS MARCO datasets that made hundreds of thousands of human annotated training labels available for both passage and document ranking tasks. In addition, this year we also leverage both the refreshed passage and document collections that were released last year leading to a nearly $16$ times increase in the size of the passage collection and nearly four times increase in the document collection size. Unlike previous years, in 2022 we mainly focused on constructing a more complete test collection for the passage retrieval task, which has been the primary focus of the track. The document ranking task was kept as a secondary task, where document-level labels were inferred from the passage-level labels. Our analysis shows that similar to previous years, deep neural ranking models that employ large scale pretraining continued to outperform traditional retrieval methods. Due to the focusing our judging resources on passage judging, we are more confident in the quality of this year's queries and judgments, with respect to our ability to distinguish between runs and reuse the dataset in future. We also see some surprises in overall outcomes. Some top-performing runs did not do dense retrieval. Runs that did single-stage dense retrieval were not as competitive this year as they were last year.

Motivation & Objective

• Benchmark ad hoc retrieval methods in the large-data regime using expanded MS MARCO datasets and refreshed collections.
• Improve test collection reusability for passage ranking and propagate passage judgments to document ranking.
• Investigate the effectiveness of deep learning models, including large-scale pretraining, versus traditional retrieval approaches.
• Assess whether single-stage dense retrieval remains competitive in 2022.
• Provide detailed analysis of submission types, evaluation metrics, and dataset design decisions.

Proposed method

• Use MS MARCO v2 datasets for passage and document ranking tasks.
• Focus evaluation on passage ranking as the primary task and infer document rankings from passage labels.
• Introduce query sampling changes to increase difficulty and reduce score saturation; judge only passages and propagate labels to documents.
• Cluster near-duplicate passages to judge one representative per class and propagate labels within the class.
• Evaluate runs with standard IR metrics (NDCG@10, NCG@100, AP) using NIST judgments; primary metric is NDCG@10.
• Categorize submissions into trad, nn, nnlm; analyze prevalence and fullrank/dense retrieval usage.

Experimental results

Research questions

• RQ1 How does the 2022 deep learning track performance compare between neural ranking models with large-scale pretraining and traditional retrieval methods?
• RQ2 What is the impact of focusing judgments on passages and propagating to documents on the reliability of document-level rankings?
• RQ3 Does single-stage dense retrieval remain competitive in 2022, compared to previous years?
• RQ4 How do query sampling and dataset design choices affect discriminability and reusability of the test collection?
• RQ5 To what extent does the use of large pretrained models (nnlm) drive performance gains across passage and document ranking tasks?

Key findings

• Neural ranking with large-scale pretraining continues to outperform traditional retrieval methods across tasks.
• Some top-performing runs did not use dense retrieval, and single-stage dense retrieval was not always the strongest performer in 2022.
• Among submissions, nnlm-based approaches dominated (about 85%), with traditional methods declining, indicating convergence toward pretrained-model pipelines.
• The emphasis on passage judgments and propagation to documents supports a more reusable test collection, with a larger performance gap between neural methods and traditional methods.
• A higher proportion of full-rank submissions suggests increasing adoption of neural approaches in end-to-end retrieval settings.
• Evaluation showed improved discriminability and reuse potential of the 2022 test collection compared to 2021.