[Paper Review] BOIL: Towards Representation Change for Few-shot Learning

Jaehoon Oh, Hyung‐Jun Yoo+2 more|arXiv (Cornell University)|Aug 20, 2020

Domain Adaptation and Few-Shot Learning44 references61 citations

TL;DR

BOIL updates only the body (feature extractor) in the inner loop while freezing the head, enforcing representation change and improving cross-domain few-shot learning compared to MAML/ANIL.

ABSTRACT

Model Agnostic Meta-Learning (MAML) is one of the most representative of gradient-based meta-learning algorithms. MAML learns new tasks with a few data samples using inner updates from a meta-initialization point and learns the meta-initialization parameters with outer updates. It has recently been hypothesized that representation reuse, which makes little change in efficient representations, is the dominant factor in the performance of the meta-initialized model through MAML in contrast to representation change, which causes a significant change in representations. In this study, we investigate the necessity of representation change for the ultimate goal of few-shot learning, which is solving domain-agnostic tasks. To this aim, we propose a novel meta-learning algorithm, called BOIL (Body Only update in Inner Loop), which updates only the body (extractor) of the model and freezes the head (classifier) during inner loop updates. BOIL leverages representation change rather than representation reuse. This is because feature vectors (representations) have to move quickly to their corresponding frozen head vectors. We visualize this property using cosine similarity, CKA, and empirical results without the head. BOIL empirically shows significant performance improvement over MAML, particularly on cross-domain tasks. The results imply that representation change in gradient-based meta-learning approaches is a critical component.

Motivation & Objective

• Motivate the need for representation change (not just reuse) in gradient-based meta-learning for domain-agnostic adaptation.
• Propose BOIL, which updates only the body during inner-loop task learning to encourage representation change.
• Empirically compare BOIL with MAML/ANIL on cross-domain and fine-grained datasets.
• Analyze representation dynamics using cosine similarity and CKA to validate representation change in BOIL.
• Explore extensions to deeper networks and provide insights via ablation studies.

Proposed method

• Split the network into body (convolutional extractor) and head (classifier).
• In the inner loop, update only the body (set head learning rate to zero for inner updates).
• Formalize BOIL updates as θ_b,τ_i = θ_b − α_b ∇_{θ_b} L_{S_τ_i}(f_θ) and θ_h,τ_i = θ_h (α_h = 0).
• Compare with MAML (α_b = α_h ≠ 0) and ANIL (α_b = 0, α_h ≠ 0) to emphasize representation change.
• Provide interpretability via layer-wise cosines and Centered Kernel Alignment (CKA) to show representation change at high-level layers.
• Extend BOIL with a disconnection trick in ResNet-12 by removing the last skip connection to boost high-level representation change.

Experimental results

Research questions

• RQ1 Is representation change essential for effective meta-learning across domains, or is representation reuse sufficient?
• RQ2 Can updating only the body in the inner loop yield superior cross-domain and fine-grained few-shot performance?
• RQ3 How do representations evolve inside BOIL compared with MAML/ANIL, as measured by cosine similarity and CKA?
• RQ4 Does freezing the head in BOIL affect overfitting and enable better generalization to unseen domains?
• RQ5 Can BOIL extend to deeper backbones (e.g., ResNet-12) with architectural tweaks to preserve representation change?

Key findings

• BOIL outperforms MAML/ANIL on several benchmark and cross-domain datasets, with notable gains on Cars and CUB.
• BOIL achieves strong cross-domain adaptation, including general-to-general, general-to-specific, and bird-to-car transfers.
• Representation analysis shows BOIL exhibits representation reuse in low/mid layers and representation change in high layers, unlike MAML/ANIL which show mainly reuse.
• Freezing the head is crucial; even small head-learning rates degrade BOIL performance.
• Disconnection tricks in deeper networks (ResNet-12) further enhance BOIL by promoting high-level representation change.
• BOIL can be effectively combined with preconditioning-gradient perspectives and improves when integrated with WarpGrad.