[Paper Review] SpotTune: Transfer Learning through Adaptive Fine-tuning

Yunhui Guo, Humphrey Shi+4 more|arXiv (Cornell University)|Nov 21, 2018

Domain Adaptation and Few-Shot LearningComputer Science43 references42 citations

TL;DR

SpotTune learns per-instance fine-tuning policies for pretrained networks, routing each input through either frozen or fine-tuned layers to improve transfer learning performance across diverse datasets.

ABSTRACT

Transfer learning, which allows a source task to affect the inductive bias of the target task, is widely used in computer vision. The typical way of conducting transfer learning with deep neural networks is to fine-tune a model pre-trained on the source task using data from the target task. In this paper, we propose an adaptive fine-tuning approach, called SpotTune, which finds the optimal fine-tuning strategy per instance for the target data. In SpotTune, given an image from the target task, a policy network is used to make routing decisions on whether to pass the image through the fine-tuned layers or the pre-trained layers. We conduct extensive experiments to demonstrate the effectiveness of the proposed approach. Our method outperforms the traditional fine-tuning approach on 12 out of 14 standard datasets.We also compare SpotTune with other state-of-the-art fine-tuning strategies, showing superior performance. On the Visual Decathlon datasets, our method achieves the highest score across the board without bells and whistles.

Motivation & Objective

• Motivate improvement over standard fine-tuning by coupling per-instance adaptation decisions with transfer learning from pretrained nets.
• Propose a lightweight policy network that decides, for each input, which residual blocks to fine-tune vs freeze.
• Introduce a training mechanism using Gumbel Softmax to enable differentiable learning of discrete routing decisions.
• Explore a global-policy variant to constrain fine-tuning to a fixed set of blocks to reduce parameters.

Proposed method

• Represent each residual block with a frozen pretrained version and a trainable replicated block initialized from the pretrained block.
• Learn a per-instance binary decision I_l(x) (freeze or fine-tune) for each block via a policy network.
• Sample I_l(x) from a Gumbel-Softmax distribution to enable backpropagation through discrete decisions.
• Train the policy network jointly with the target task using standard classification loss, with a straight-through estimator for the Gumbel-Softmax.
• Optionally, impose a compact global policy that constrains all inputs to use the same k fine-tuned blocks and add auxiliary losses to encourage binarization.
• Provide a global-k variant that reduces parameters by discarding untouched pretrained blocks.

Experimental results

Research questions

• RQ1 Can per-instance routing of inputs through pretrained vs fine-tuned blocks improve transfer learning performance over uniform fine-tuning strategies?
• RQ2 Does a global policy that fixes a subset of blocks achieve competitive accuracy with fewer parameters?
• RQ3 How does SpotTune compare to state-of-the-art fine-tuning methods on diverse datasets and the Visual Decathlon benchmark?
• RQ4 What insights into learned fine-tuning policies emerge when visualizing block usage across tasks?

Key findings

• SpotTune outperforms standard fine-tuning on 12 of 14 datasets tested.
• The method achieves highest Visual Decathlon score among compared approaches without modifying architecture.
• A per-instance policy yields dataset-specific and example-specific routing decisions, enabling better feature reuse and adaptation.
• A compact global-k variant reduces parameters while maintaining strong performance, outperforming some last-k fine-tuning baselines.
• L2-SP enhances fine-tuning performance but is outperformed by SpotTune and is complementary to it.
• Visualizations show diverse fine-tuning policies across datasets, suggesting non-contiguous and input-dependent fine-tuning is beneficial.