[Paper Review] Sample-based Regularization: A Transfer Learning Strategy Toward Better Generalization

Yunho Jeon, Yong-Seok Choi+4 more|arXiv (Cornell University)|Jul 10, 2020

Domain Adaptation and Few-Shot Learning31 references4 citations

TL;DR

This paper proposes Sample-based Regularization (SBR), a novel transfer learning strategy that enhances generalization in few-shot learning by regularizing feature embeddings of same-class samples to be more similar, without relying on the source model during training. SBR outperforms existing methods across multiple datasets and sampling rates, achieving state-of-the-art accuracy with minimal inductive bias from the source task.

ABSTRACT

Training a deep neural network with a small amount of data is a challenging problem as it is vulnerable to overfitting. However, one of the practical difficulties that we often face is to collect many samples. Transfer learning is a cost-effective solution to this problem. By using the source model trained with a large-scale dataset, the target model can alleviate the overfitting originated from the lack of training data. Resorting to the ability of generalization of the source model, several methods proposed to use the source knowledge during the whole training procedure. However, this is likely to restrict the potential of the target model and some transferred knowledge from the source can interfere with the training procedure. For improving the generalization performance of the target model with a few training samples, we proposed a regularization method called sample-based regularization (SBR), which does not rely on the source's knowledge during training. With SBR, we suggested a new training framework for transfer learning. Experimental results showed that our framework outperformed existing methods in various configurations.

Motivation & Objective

• To address the challenge of overfitting in deep learning when training with small datasets.
• To improve generalization performance in transfer learning without relying on the source model as a regularization reference during training.
• To explore alternative regularization mechanisms that are more aligned with the target task's intrinsic structure.
• To reduce interference from potentially irrelevant source knowledge that may hinder target task optimization.
• To develop a method that enhances feature learning through pairwise sample similarity, promoting robustness in low-data regimes.

Proposed method

• Proposes Sample-based Regularization (SBR), which regularizes the feature embeddings of samples from the same class to maximize their similarity.
• Applies SBR on the feature extractor's output, using a contrastive loss to minimize the distance between same-class features.
• Decouples the classifier and feature extractor, applying cross-entropy loss only to the classifier and SBR loss only to the feature extractor to reduce interference.
• Uses a pairwise similarity loss that encourages the network to learn a metric space where same-class samples are closer, akin to metric learning.
• Introduces a training framework where the source model is used only for initialization, not for regularization during optimization.
• Employs a two-branch loss strategy: cross-entropy for classification and SBR for feature-level consistency within classes.

Experimental results

Research questions

• RQ1 Can regularization based on intra-class sample similarity improve generalization in few-shot transfer learning?
• RQ2 Does avoiding reliance on the source model during training lead to better performance on target tasks with limited data?
• RQ3 How does SBR compare to existing transfer learning methods that use the source model as a reference for regularization?
• RQ4 Can pairwise similarity regularization generalize better than one-hot target supervision in low-data regimes?
• RQ5 Does SBR maintain performance gains when the source model is changed (e.g., from ImageNet to Places365)?

Key findings

• SBR achieved 74.91% accuracy on CUB at 100% sampling rate using Places365 as a source, outperforming the best baseline (67.39%) by 7.52 percentage points.
• On the Cars dataset, SBR achieved 90.59% accuracy at 100% sampling rate, surpassing the next best method (88.61%) by 1.98 percentage points.
• On the Aircraft dataset, SBR achieved 85.04% accuracy at 100% sampling rate, exceeding the baseline (78.98%) by 6.06 percentage points.
• On the Flowers dataset, SBR achieved 94.73% accuracy at 100% sampling rate, outperforming the baseline (92.14%) by 2.59 percentage points.
• SBR consistently outperformed all baselines—including L2-SP, DELTA, and BSS variants—across all datasets and sampling rates, demonstrating robustness and generalization.
• The method showed strong performance even with low data availability, achieving 67.47% on Flowers at 15% sampling rate, significantly above the baseline of 59.27%.