[Paper Review] Explicit Inductive Bias for Transfer Learning with Convolutional Networks

Xuhong Li, Yves Grandvalet+1 more|arXiv (Cornell University)|Feb 5, 2018

Domain Adaptation and Few-Shot LearningComputer Science149 citations

TL;DR

The paper introduces explicit regularizers that bias fine-tuning toward the pre-trained starting point, showing that L2-SP (and L2-SP-Fisher) consistently outperform standard L2 fine-tuning across several transfer learning tasks.

ABSTRACT

In inductive transfer learning, fine-tuning pre-trained convolutional networks substantially outperforms training from scratch. When using fine-tuning, the underlying assumption is that the pre-trained model extracts generic features, which are at least partially relevant for solving the target task, but would be difficult to extract from the limited amount of data available on the target task. However, besides the initialization with the pre-trained model and the early stopping, there is no mechanism in fine-tuning for retaining the features learned on the source task. In this paper, we investigate several regularization schemes that explicitly promote the similarity of the final solution with the initial model. We show the benefit of having an explicit inductive bias towards the initial model, and we eventually recommend a simple $L^2$ penalty with the pre-trained model being a reference as the baseline of penalty for transfer learning tasks.

Motivation & Objective

• Motivate the use of explicit inductive bias to preserve knowledge from pre-trained CNNs during fine-tuning.
• Propose and compare several regularizers that reference the pre-trained parameters during transfer learning.
• Evaluate regularizers on several source-target task pairs to assess benefits over standard fine-tuning.
• Provide practical recommendations for baseline regularization in inductive transfer learning with CNNs.

Proposed method

• Define regularized objective by adding a penalty term to the standard loss, with the pre-trained weights w0 as reference.
• Introduce L2-SP: Omega(w) = (alpha/2) * ||w - w0||^2 as a baseline penalty.
• Extend to partial sharing with separate penalties for shared vs. new parameters when target architecture differs from source.
• Explore L2-SP-Fisher: a Fisher-information-weighted variant of L2-SP that preserves source-task sensitivity.
• Investigate L1-SP and Group-Lasso-SP variants to encourage parameter freezing at units or groups, including GL-SP-Fisher.
• Conduct experiments with ResNet across multiple source/target pairs (ImageNet/Places 365 to Caltech 256 MIT Indoors 67 Stanford Dogs 120).

Experimental results

Research questions

• RQ1 Does an explicit inductive bias toward the pre-trained starting point improve transfer learning performance over standard fine-tuning?
• RQ2 How do L2-SP, L2-SP-Fisher, and other SP-based penalties compare in accuracy and stability across different target tasks?
• RQ3 Is there a practical baseline penalty that consistently outperforms standard weight decay for transfer learning?
• RQ4 What is the impact of freezing layers versus applying SP penalties on transfer performance?
• RQ5 Do Fisher-weighted penalties provide a meaningful advantage in inductive transfer scenarios?

Key findings

• L2-SP and L2-SP-Fisher consistently improve over standard L2 fine-tuning across all target tasks.
• The improvements from L2-SP are more pronounced when target data are scarce.
• L1-SP and Group-Lasso-SP variants perform poorly or are less favorable than L2-based SP methods in these transfer settings.
• Fisher-information based SP does not significantly outperform simple L2-SP in target-task performance, though it reduces forgetting in lifelong-learning-like scenarios.
• Freezing layers is often less effective than L2-SP fine-tuning for preserving pre-trained knowledge while adapting to the target task.