[Paper Review] Ranking Distance Calibration for Cross-Domain Few-Shot Learning

Li Pan, Shaogang Gong+2 more|arXiv (Cornell University)|Dec 1, 2021

Domain Adaptation and Few-Shot Learning41 references4 citations

TL;DR

This paper proposes Ranking Distance Calibration (RDC) for cross-domain few-shot learning (CD-FSL), which improves distance-based classification by re-ranking k-reciprocal neighbors in the target domain and calibrating distances via a non-linear subspace. RDC-FT further fine-tunes the model using a Kullback-Leibler divergence loss to embed calibrated distances, achieving state-of-the-art performance across eight diverse target domains with consistent gains over baseline methods.

ABSTRACT

Recent progress in few-shot learning promotes a more realistic cross-domain setting, where the source and target datasets are from different domains. Due to the domain gap and disjoint label spaces between source and target datasets, their shared knowledge is extremely limited. This encourages us to explore more information in the target domain rather than to overly elaborate training strategies on the source domain as in many existing methods. Hence, we start from a generic representation pre-trained by a cross-entropy loss and a conventional distance-based classifier, along with an image retrieval view, to employ a re-ranking process for calibrating a target distance matrix by discovering the reciprocal k-nearest neighbours within the task. Assuming the pre-trained representation is biased towards the source, we construct a non-linear subspace to minimise task-irrelevant features therewithin while keep more transferrable discriminative information by a hyperbolic tangent transformation. The calibrated distance in this target-aware non-linear subspace is complementary to that in the pre-trained representation. To impose such distance calibration information onto the pre-trained representation, a Kullback-Leibler divergence loss is employed to gradually guide the model towards the calibrated distance-based distribution. Extensive evaluations on eight target domains show that this target ranking calibration process can improve conventional distance-based classifiers in few-shot learning.

Motivation & Objective

• To address the challenge of domain shift and disjoint label spaces in cross-domain few-shot learning (CD-FSL), where source and target domains differ significantly in distribution and class labels.
• To reduce bias in pre-trained representations that are optimized for the source domain but poorly generalize to the target domain.
• To improve distance-based classification in CD-FSL by discovering more reliable positive samples through target-domain re-ranking using k-reciprocal neighbors.
• To develop a method that is orthogonal and complementary to existing model learning approaches, enabling flexible integration with fine-tuning or non-fine-tuning frameworks.
• To calibrate the distance matrix in a way that enhances discriminative power by minimizing task-irrelevant features while preserving transferable knowledge.

Proposed method

• The method begins with a pre-trained representation using cross-entropy loss and a conventional distance-based classifier, treating CD-FSL as an image retrieval task.
• It applies a k-reciprocal neighbor discovery process to expand the k-nearest neighbor list in the target domain, improving robustness of initial rankings.
• A Jaccard distance is computed between the original and expanded ranking lists to re-compute pairwise distances, forming a more accurate distance matrix.
• A non-linear subspace is learned via a hyperbolic tangent transformation to project the source representation, minimizing transfer of task-irrelevant features while preserving discriminative information.
• The calibrated distance matrix is fused with the original pre-trained distance matrix to form a robust, hybrid distance measure.
• RDC-FT fine-tunes the model using a Kullback-Leibler divergence loss to gradually align the feature distribution with the calibrated distance-based distribution.

Experimental results

Research questions

• RQ1 Can re-ranking based on k-reciprocal neighbors improve distance calibration in cross-domain few-shot learning?
• RQ2 Does learning a non-linear, task-adaptive subspace reduce bias from source-domain pre-trained representations in CD-FSL?
• RQ3 Can the calibrated distance matrix be effectively integrated into the model via a differentiable loss to improve generalization?
• RQ4 How does the proposed method compare to state-of-the-art approaches across diverse, unseen target domains?
• RQ5 Can the method be flexibly combined with other training strategies, such as data augmentation, to further improve performance?

Key findings

• RDC-FT achieves state-of-the-art performance across eight target domains, with consistent improvements over baseline methods including NPC and data-augmentation baselines.
• On the 5-way 1-shot setting, RDC-FT improved accuracy by 12.47% on CropDisease (from 71.94% to 84.73%) when combined with data augmentation.
• The KL divergence loss outperformed MSE and cross-entropy losses in most domains, particularly on Places, Plantae, and EuroSAT, indicating the benefit of softened distance distributions.
• t-SNE visualizations show that RDC-FT reduces intra-class variation and increases inter-class margins, leading to more discriminative representations.
• RDC consistently improved performance when combined with a data augmentation method, demonstrating its generalization and compatibility with other training strategies.
• Failure cases in RDC were corrected by RDC-FT, confirming that the fine-tuning process effectively embeds calibration information into the representation space.