[论文解读] Can Old TREC Collections Reliably Evaluate Modern Neural Retrieval Models?

Ellen M. Voorhees, Ian Soboroff+1 more|arXiv (Cornell University)|Jan 26, 2022

Explainable Artificial Intelligence (XAI)被引用 4

一句话总结

本研究评估了1999年构建的TREC-8即席测试集是否仍适用于现代神经检索模型的评估。尽管神经模型检索到了大量此前未被标注的文档——其中绝大多数为无关文档——但使用原始相关性标注和扩展相关性标注的运行结果排名几乎完全一致，肯德尔等级相关系数τ超过0.99，证明该测试集对现代系统仍具持久的可重用性。

ABSTRACT

Neural retrieval models are generally regarded as fundamentally different from the retrieval techniques used in the late 1990's when the TREC ad hoc test collections were constructed. They thus provide the opportunity to empirically test the claim that pooling-built test collections can reliably evaluate retrieval systems that did not contribute to the construction of the collection (in other words, that such collections can be reusable). To test the reusability claim, we asked TREC assessors to judge new pools created from new search results for the TREC-8 ad hoc collection. These new search results consisted of five new runs (one each from three transformer-based models and two baseline runs that use BM25) plus the set of TREC-8 submissions that did not previously contribute to pools. The new runs did retrieve previously unseen documents, but the vast majority of those documents were not relevant. The ranking of all runs by mean evaluation score when evaluated using the official TREC-8 relevance judgment set and the newly expanded relevance set are almost identical, with Kendall's tau correlations greater than 0.99. Correlations for individual topics are also high. The TREC-8 ad hoc collection was originally constructed using deep pools over a diverse set of runs, including several effective manual runs. Its judgment budget, and hence construction cost, was relatively large. However, it does appear that the expense was well-spent: even with the advent of neural techniques, the collection has stood the test of time and remains a reliable evaluation instrument as retrieval techniques have advanced.

研究动机与目标

• 测试1999年构建的TREC-8即席测试集是否仍适用于评估现代神经检索模型。
• 评估神经模型是否检索到大量此前未被标注的相关文档，从而实质性地改变系统排名。
• 评估基于聚类的测试集对未参与其构建的系统是否具有可重用性。
• 确定原始TREC-8测试集中高质量的人工标注运行结果是否对维持评估可靠性至关重要。

提出的方法

• 使用五项新运行结果构建新聚类：三种基于Transformer的模型（TCT-ColBERT、monoBERT、uniCOIL）和两种基于BM25的基线模型，以及此前未被标注的TREC-8提交结果。
• 从TREC评估员处收集对新运行结果检索到但此前未被标注的文档的新相关性标注。
• 使用官方TREC-8相关性标注集和扩展标注集评估系统性能，比较平均得分和排名。
• 计算使用原始和扩展qrels时系统排名之间的肯德尔等级相关系数τ，以衡量相对性能的稳定性。
• 重建TREC-8测试集，不包含人工运行结果，以检验人工运行结果对评估可靠性的影响。
• 分析排除人工运行贡献后丢失的相关文档比例，并评估其对系统排名的影响。

实验结果

研究问题

• RQ1 现代神经检索模型是否在TREC-8测试集中检索到大量此前未被标注的相关文档？
• RQ2 将神经模型输出的新相关性标注纳入后，是否显著改变了在TREC-8上评估的检索系统之间的相对排名？
• RQ3 人工运行结果对TREC-8测试集的可靠性与可重用性有多关键？
• RQ4 若仅基于自动运行构建测试集（不包含人工运行结果），其质量是否仍足以可靠评估现代系统？

主要发现

• 绝大多数由新神经模型检索到但此前未被标注的文档被判定为不相关，仅有极小部分为新发现的相关文档。
• 使用原始和扩展相关性标注时，系统排名之间的肯德尔等级相关系数τ在P@10和MAP上均超过0.99，表明相对性能几乎完全一致。
• 即使在原始聚类构建中排除人工运行结果，排名相关性依然极高（P@10为0.9964，MAP为0.9818），表明对人工标注损失具有强鲁棒性。
• 在原始仅由人工运行检索到的1,131篇相关文档中，仅有100篇被新运行结果重新检索到，其中TCT-ColBERT检索到50篇，monoBERT检索到22篇，uniCOIL检索到17篇。
• BM25+RM3基线仅检索到两篇仅由人工运行发现的相关文档，表明其与神经模型检索模式重叠度极低。
• 尽管丢失了超过10%的已知相关文档，系统排名仍保持高度稳定，表明TREC-8测试集对现代神经检索模型而言仍具可靠的评估能力。