[Paper Review] Can Old TREC Collections Reliably Evaluate Modern Neural Retrieval Models?

Ellen M. Voorhees, Ian Soboroff+1 more|arXiv (Cornell University)|Jan 26, 2022

Explainable Artificial Intelligence (XAI)4 citations

TL;DR

This study evaluates whether the TREC-8 ad hoc test collection remains reliable for assessing modern neural retrieval models. Despite neural models retrieving many previously unjudged documents—most of which were irrelevant—the rankings of runs using the original and expanded relevance judgments show near-identical performance, with Kendall’s τ correlations exceeding 0.99, demonstrating the collection's enduring reusability for modern systems.

ABSTRACT

Neural retrieval models are generally regarded as fundamentally different from the retrieval techniques used in the late 1990's when the TREC ad hoc test collections were constructed. They thus provide the opportunity to empirically test the claim that pooling-built test collections can reliably evaluate retrieval systems that did not contribute to the construction of the collection (in other words, that such collections can be reusable). To test the reusability claim, we asked TREC assessors to judge new pools created from new search results for the TREC-8 ad hoc collection. These new search results consisted of five new runs (one each from three transformer-based models and two baseline runs that use BM25) plus the set of TREC-8 submissions that did not previously contribute to pools. The new runs did retrieve previously unseen documents, but the vast majority of those documents were not relevant. The ranking of all runs by mean evaluation score when evaluated using the official TREC-8 relevance judgment set and the newly expanded relevance set are almost identical, with Kendall's tau correlations greater than 0.99. Correlations for individual topics are also high. The TREC-8 ad hoc collection was originally constructed using deep pools over a diverse set of runs, including several effective manual runs. Its judgment budget, and hence construction cost, was relatively large. However, it does appear that the expense was well-spent: even with the advent of neural techniques, the collection has stood the test of time and remains a reliable evaluation instrument as retrieval techniques have advanced.

Motivation & Objective

• To test whether the TREC-8 ad hoc test collection, built in 1999, remains reliable for evaluating modern neural retrieval models.
• To assess whether neural models retrieve a substantial number of previously unjudged relevant documents that would materially alter system rankings.
• To evaluate the reusability of pooling-based test collections for systems not involved in their construction.
• To determine whether the original TREC-8 collection’s high-quality manual runs are essential for maintaining evaluation reliability.

Proposed method

• Constructed new pools using five new runs: three transformer-based models (TCT-ColBERT, monoBERT, uniCOIL) and two BM25-based baselines, plus previously unjudged TREC-8 submissions.
• Collected new relevance judgments from TREC assessors on documents retrieved by the new runs but not previously judged.
• Evaluated system performance using official TREC-8 relevance judgments and the expanded judgment set, comparing mean scores and rankings.
• Computed Kendall’s τ correlation between system rankings using original and expanded qrels to measure stability of relative performance.
• Reconstructed the TREC-8 collection without manual runs to test the impact of manual runs on evaluation reliability.
• Analyzed the proportion of relevant documents lost when excluding manual-run contributions and assessed their impact on system ranking.

Experimental results

Research questions

• RQ1 Do modern neural retrieval models retrieve a significant number of previously unjudged relevant documents in the TREC-8 collection?
• RQ2 Does the inclusion of new relevance judgments from neural model outputs substantially alter the relative ranking of retrieval systems evaluated on TREC-8?
• RQ3 How critical are manual runs to the reliability and reusability of the TREC-8 test collection for modern neural models?
• RQ4 Would a test collection built solely from automatic runs (without manual runs) maintain sufficient quality for reliable evaluation of modern systems?

Key findings

• The vast majority of documents retrieved by new neural runs but not previously judged were found to be not relevant, with only a small fraction being newly relevant.
• Kendall’s τ correlation between system rankings using original and expanded relevance judgments exceeded 0.99 for both P@10 and MAP, indicating near-identical relative performance.
• Even when manual runs were excluded from the original pool construction, the correlation between rankings remained very high (0.9964 for P@10 and 0.9818 for MAP), showing robustness to loss of manual judgments.
• Only 100 of the 1,131 relevant documents originally retrieved exclusively by manual runs were recovered by the new runs, with TCT-ColBERT recovering 50, monoBERT 22, and uniCOIL 17.
• The BM25+RM3 baseline recovered only two of the manual-only relevant documents, indicating limited overlap with neural model retrieval patterns.
• Despite the loss of over 10% of known relevant documents, system rankings remained highly stable, suggesting that the TREC-8 collection remains a reliable evaluation instrument for modern neural models.