[Paper Review] Overview of the TREC 2019 deep learning track

Nick Craswell, Bhaskar Mitra+3 more|arXiv (Cornell University)|Mar 17, 2020

Topic Modeling16 references71 citations

TL;DR

The paper introduces the TREC 2019 Deep Learning Track with two ad hoc retrieval tasks, large MS MARCO–based training data, blind evaluation, and a comparison across nnlm, nn, and traditional IR methods, including end-to-end vs reranking analysis.

ABSTRACT

The Deep Learning Track is a new track for TREC 2019, with the goal of studying ad hoc ranking in a large data regime. It is the first track with large human-labeled training sets, introducing two sets corresponding to two tasks, each with rigorous TREC-style blind evaluation and reusable test sets. The document retrieval task has a corpus of 3.2 million documents with 367 thousand training queries, for which we generate a reusable test set of 43 queries. The passage retrieval task has a corpus of 8.8 million passages with 503 thousand training queries, for which we generate a reusable test set of 43 queries. This year 15 groups submitted a total of 75 runs, using various combinations of deep learning, transfer learning and traditional IR ranking methods. Deep learning runs significantly outperformed traditional IR runs. Possible explanations for this result are that we introduced large training data and we included deep models trained on such data in our judging pools, whereas some past studies did not have such training data or pooling.

Motivation & Objective

• Provide large reusable training and test datasets for deep learning and traditional ranking methods in a large data regime.
• Conduct a rigorous blind single-shot evaluation to compare ranking methods.
• Study end-to-end retrieval versus reranking in practical IR deployments.
• Compare neural language models, neural network approaches, and traditional IR baselines under consistent evaluation.
• Explore the interaction between deep learning methods and traditional indexing in large-scale retrieval.

Proposed method

• Two tasks (document retrieval and passage retrieval) with end-to-end and reranking participation styles (fullrank vs rerank).
• Large training sets derived from MS MARCO with transferred labels from passages to documents.
• Categorization of runs into nnlm (pre-trained neural language models like BERT/XLNet), nn (neural nets), and trad (traditional IR).
• Evaluation using NDCG@10 as the main metric, with NCG@k as a secondary analysis to assess recall of relevant results.
• Use of 43 reusable test queries per task and blind judging by NIST, with pooled judgments and HiCAL-assisted expansion.
• Visualization of inter-run similarity using 43-dimensional NDCG vectors reduced by t-SNE to analyze clustering by model type and group.

Experimental results

Research questions

• RQ1 Do deep learning models (nnlm) outperform traditional IR methods (trad) in large-data ad hoc retrieval tasks?
• RQ2 How do nnlm, nn, and trad methods compare across document retrieval and passage retrieval tasks?
• RQ3 What is the impact of end-to-end retrieval (fullrank) versus reranking (rerank) on retrieval effectiveness?
• RQ4 Are MS MARCO–based training labels transferable and predictive for TREC-style evaluations using NIST labels?

Key findings

• nnlm runs outperform nn and trad runs in both document and passage retrieval tasks. The best nnlm vs best trad gap in NDCG@10 is 29.4% for document retrieval and 37.4% for passage retrieval.
• End-to-end fullrank submissions do not consistently outperform rerank submissions; the best observed difference in NDCG@10 is 0.9% (document) and 3.6% (passage) in favor of fullrank.
• For document retrieval, the highest NCG@100 is achieved by a BM25+RM3 setup with doc2query enhancements, showing a 22.9% improvement over the 100-candidate baseline.
• t-SNE analysis shows runs cluster by group and by model-type category (nnlm, nn, trad), indicating meaningful similarities among methods.
• Cross-evaluation with MS MARCO vs NIST labels shows general agreement in run ordering, supporting MS MARCO results as indicative of traditional TREC performance.