[Paper Review] Fine-Grained Distillation for Long Document Retrieval

Yucheng Zhou, Tao Shen+5 more|arXiv (Cornell University)|Dec 20, 2022

Image Retrieval and Classification Techniques4 citations

TL;DR

This paper proposes Fine-Grained Distillation (FGD), a novel knowledge distillation framework for long-document retrieval that mitigates granularity mismatch by training a bi-encoder to align with multi-granular representations from a cross-encoder. By using global-consistent granularity embeddings and local-coordinated distillation across sentence and passage levels during training, FGD achieves state-of-the-art performance on MS-Marco and TREC 2019 benchmarks without affecting inference efficiency.

ABSTRACT

Long document retrieval aims to fetch query-relevant documents from a large-scale collection, where knowledge distillation has become de facto to improve a retriever by mimicking a heterogeneous yet powerful cross-encoder. However, in contrast to passages or sentences, retrieval on long documents suffers from the scope hypothesis that a long document may cover multiple topics. This maximizes their structure heterogeneity and poses a granular-mismatch issue, leading to an inferior distillation efficacy. In this work, we propose a new learning framework, fine-grained distillation (FGD), for long-document retrievers. While preserving the conventional dense retrieval paradigm, it first produces global-consistent representations crossing different fine granularity and then applies multi-granular aligned distillation merely during training. In experiments, we evaluate our framework on two long-document retrieval benchmarks, which show state-of-the-art performance.

Motivation & Objective

• Address the performance gap between bi-encoders and cross-encoders in long-document retrieval due to scope hypothesis and granularity mismatch.
• Overcome the limitation of existing distillation methods that fail on long documents because of structural and semantic heterogeneity across topics.
• Develop a training-time framework that enables effective knowledge transfer from a heterogeneous cross-encoder to a bi-encoder without altering inference efficiency.
• Ensure consistent representations across fine-grained units (e.g., sentences, passages) while preserving global document-level semantics.
• Enable generalization across diverse bi-encoder and cross-encoder architectures through a unified distillation strategy.

Proposed method

• Introduce global-consistent granularity embedding to dynamically contextualize representations at multiple granularities (sentence, passage) within a long document.
• Propose local-coordinating score distillation that transfers relevance score distributions from the cross-encoder to the bi-encoder at multiple granularities during training.
• Design a hierarchical negative mining strategy that samples hard negatives from top-level (document) to fine-grained levels (sentence, passage) to improve representation learning.
• Apply multi-granular distillation only during training, preserving single-vector inference in deployment to maintain efficiency.
• Use a bi-encoder for inference with fixed, single-vector document embeddings, while leveraging multi-granular supervision during training to improve generalization.
• Combine distillation with contrastive learning to enhance the bi-encoder’s robustness and reduce overfitting.

Experimental results

Research questions

• RQ1 Can knowledge distillation from a cross-encoder to a bi-encoder be effectively applied to long documents despite their multi-topic nature?
• RQ2 Does multi-granular distillation improve bi-encoder performance on long documents compared to document-level distillation?
• RQ3 Can global-consistent granularity embeddings improve alignment between fine-grained and global representations in long documents?
• RQ4 How does hierarchical negative mining affect the quality of distillation in long-document retrieval?
• RQ5 Is the proposed FGD framework compatible with various bi-encoder and cross-encoder architectures?

Key findings

• FGD achieves state-of-the-art performance on both MS-Marco and TREC 2019 long-document retrieval benchmarks, outperforming existing distillation-based methods.
• The method achieves significant performance gains—over 1% improvement in retrieval metrics—on long documents, in contrast to minimal gains (<0.1%) from brute-force distillation in pilot experiments.
• The framework maintains inference efficiency by applying multi-granular distillation only during training, preserving the standard bi-encoder inference pipeline.
• Global-consistent granularity embeddings effectively reduce cross-granularity conflict and improve alignment between fine-grained and global representations.
• Hierarchical negative mining enhances distillation by providing more informative hard negatives across multiple granularities, improving model generalization.
• The framework is compatible with various bi-encoder and cross-encoder architectures, demonstrating strong generalization across different model choices.