[Paper Review] SB-MTL: Score-based Meta Transfer-Learning for Cross-Domain Few-Shot Learning

John Cai, Bill Cai+1 more|arXiv (Cornell University)|Dec 3, 2020

Domain Adaptation and Few-Shot Learning45 references4 citations

TL;DR

This paper proposes Score-based Meta Transfer-Learning (SB-MTL), a novel method that combines first-order MAML optimization with a score-based Graph Neural Network (GNN) to improve cross-domain few-shot learning under sharp domain shifts. By mapping pre-softmax classification scores to a metric space and using GNNs to refine predictions via label propagation, SB-MTL achieves state-of-the-art accuracy on the BSCD-FSL benchmark, outperforming prior meta-learning and transfer-learning methods by 14.28% and 5.93% on average, respectively.

ABSTRACT

While many deep learning methods have seen significant success in tackling the problem of domain adaptation and few-shot learning separately, far fewer methods are able to jointly tackle both problems in Cross-Domain Few-Shot Learning (CD-FSL). This problem is exacerbated under sharp domain shifts that typify common computer vision applications. In this paper, we present a novel, flexible and effective method to address the CD-FSL problem. Our method, called Score-based Meta Transfer-Learning (SB-MTL), combines transfer-learning and meta-learning by using a MAML-optimized feature encoder and a score-based Graph Neural Network. First, we have a feature encoder with specific layers designed to be fine-tuned. To do so, we apply a first-order MAML algorithm to find good initializations. Second, instead of directly taking the classification scores after fine-tuning, we interpret the scores as coordinates by mapping the pre-softmax classification scores onto a metric space. Subsequently, we apply a Graph Neural Network to propagate label information from the support set to the query set in our score-based metric space. We test our model on the Broader Study of Cross-Domain Few-Shot Learning (BSCD-FSL) benchmark, which includes a range of target domains with highly varying dissimilarity to the miniImagenet source domain. We observe significant improvements in accuracy across 5, 20 and 50 shot, and on the four target domains. In terms of average accuracy, our model outperforms previous transfer-learning methods by 5.93% and previous meta-learning methods by 14.28%.

Motivation & Objective

• Address the challenge of cross-domain few-shot learning (CD-FSL) under sharp domain shifts, where existing methods fail due to distributional shifts between source and target domains.
• Overcome the limitations of current meta-learning and transfer-learning approaches, which underperform on highly dissimilar domains such as satellite, medical, and natural images.
• Develop a unified framework that effectively combines the strengths of meta-learning (fast adaptation) and transfer-learning (feature transfer) for robust generalization across domains.
• Design a metric space based on classification scores rather than raw features to enable domain-agnostic representation learning.
• Demonstrate that a score-based GNN can correct asymmetric confusion patterns in initial predictions, improving robustness and accuracy in low-shot, out-of-distribution settings.

Proposed method

• Use a deep residual network with fine-tunable layers, optimized via a first-order MAML algorithm to learn good initial weights for fast adaptation.
• Map pre-softmax classification scores from the fine-tuned model onto a learned metric space using a linear projection layer, creating a score-based representation.
• Apply a Graph Neural Network (GNN) to propagate label information from support-set images to query images in the score-based metric space, enabling message passing based on predicted scores and true labels.
• Construct a graph where nodes represent support and query images, and edges encode similarity in classification scores, allowing the GNN to refine predictions through neighborhood aggregation.
• Integrate data augmentation during inference to further boost performance, especially in low-shot regimes.
• Train end-to-end using episodic training, where each episode samples a few classes with support and query sets, optimizing for few-shot generalization.

Experimental results

Research questions

• RQ1 Can a meta-learning framework effectively generalize to highly dissimilar target domains in cross-domain few-shot learning?
• RQ2 How does using pre-softmax classification scores as input to a GNN improve performance compared to direct prediction or feature-based metric learning?
• RQ3 To what extent can a score-based metric space reduce overfitting and correct for asymmetric confusion patterns in low-shot settings?
• RQ4 Does combining MAML-based fine-tuning with a GNN-based reclassification module outperform standalone meta-learning or transfer-learning baselines on the BSCD-FSL benchmark?
• RQ5 How does the performance of the score-based GNN vary across domains with different degrees of similarity to the source domain?

Key findings

• SB-MTL achieves an average accuracy of 74.06% on the BSCD-FSL benchmark, significantly outperforming previous state-of-the-art meta-learning methods by 14.28% and transfer-learning methods by 5.93%.
• On 50-shot settings, SB-MTL achieves accuracies on EuroSAT and CropDisease that are comparable to or higher than those of typical supervised deep learning models.
• The addition of the score-based GNN improves accuracy over the baseline Linear MTL (first-order MAML only), with the largest gains observed on CropDisease and EuroSAT, where domain shifts are moderate.
• Asymmetry measures show that SB-MTL reduces confusion matrix asymmetry by 63.95% on CropDisease and 58.12% on EuroSAT compared to Linear MTL, indicating better correction of biased initial predictions.
• The score-based GNN is most effective in domains closer to the source domain, as evidenced by higher accuracy gains on EuroSAT and CropDisease than on ISIC and ChestX.
• Data augmentation alone cannot explain the performance gap; the score-based GNN provides a substantial and consistent improvement beyond augmentation, especially in high-domain-shift settings.