[论文解读] Pretrained Transformers for Text Ranking: BERT and Beyond

Jimmy Lin, Rodrigo Nogueira+1 more|arXiv (Cornell University)|Oct 13, 2020

Topic Modeling参考文献 320被引用 107

一句话总结

对预训练变换模型，特别是 BERT，在文本排序中的应用进行综述，涵盖再排序和密集检索、长文本处理、效率权衡与未来方向。

ABSTRACT

The goal of text ranking is to generate an ordered list of texts retrieved from a corpus in response to a query. Although the most common formulation of text ranking is search, instances of the task can also be found in many natural language processing applications. This survey provides an overview of text ranking with neural network architectures known as transformers, of which BERT is the best-known example. The combination of transformers and self-supervised pretraining has been responsible for a paradigm shift in natural language processing (NLP), information retrieval (IR), and beyond. In this survey, we provide a synthesis of existing work as a single point of entry for practitioners who wish to gain a better understanding of how to apply transformers to text ranking problems and researchers who wish to pursue work in this area. We cover a wide range of modern techniques, grouped into two high-level categories: transformer models that perform reranking in multi-stage architectures and dense retrieval techniques that perform ranking directly. There are two themes that pervade our survey: techniques for handling long documents, beyond typical sentence-by-sentence processing in NLP, and techniques for addressing the tradeoff between effectiveness (i.e., result quality) and efficiency (e.g., query latency, model and index size). Although transformer architectures and pretraining techniques are recent innovations, many aspects of how they are applied to text ranking are relatively well understood and represent mature techniques. However, there remain many open research questions, and thus in addition to laying out the foundations of pretrained transformers for text ranking, this survey also attempts to prognosticate where the field is heading.

研究动机与目标

• 综合说明变换模型在文本排序中的应用，面向从业者和研究人员。
• 将技术分为多阶段再排序和密集检索方法。
• 突出处理长文档的方法以及有效性与效率之间的权衡。
• 调查当前现状并识别未解的研究问题和未来方向。

提出的方法

• 描述使用变换模型进行再排序的多阶段架构（如 monoBERT、Birch、PARADE、CEDR）。
• 讨论使用变换器双编码器的密集检索（如 DPR、ANCE）以及晚期交互模型（如 ColBERT）。
• 涵盖知识蒸馏和模型变体（如 TK、TKL、CK、monoT5）。
• 解释查询/文档扩展与重新加权技术（doc2query、DeepCT、HDCT、CEQE）。
• 涉及长文本处理与效率考量（延迟、索引大小）及实际部署关注点。

实验结果

研究问题

• RQ1 如何在跨领域的文本排序任务中有效应用预训练变换模型？
• RQ2 在变换模型的约束下，哪些策略最有效地处理长文档以用于排序？
• RQ3 基于变换模型的系统中，排序效果与效率之间的权衡是什么？
• RQ4 在将变换模型应用于文本排序方面，存在哪些尚待解决的研究问题？

主要发现

• 基于变换模型在文本排序领域与任务中提供高质量的结果。
• BERT 在相关性分类方面，以及作为再排序架构的基础方面，均表现出强大性能。
• 在长文本处理以及在准确性、延迟和索引大小之间实现平衡方面存在有效策略。
• 密集表示和近似最近邻搜索在某些设置中实现直接的单阶段排序。
• 变体和蒸馏技术在发挥变换模型优势的同时提供不同的效率–效果权衡。
• 该领域核心技术已成熟，但仍存在持续的开放问题和未来工作方向。