[Paper Review] Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions

Han-Jia Ye, Hexiang Hu+2 more|arXiv (Cornell University)|Dec 10, 2018

Domain Adaptation and Few-Shot LearningComputer Science63 references81 citations

TL;DR

The paper proposes task-specific embedding adaptation for few-shot learning using set-to-set functions, with Transformer-based FEAT as the final model, achieving state-of-the-art results on several benchmarks and extended settings.

ABSTRACT

Learning with limited data is a key challenge for visual recognition. Many few-shot learning methods address this challenge by learning an instance embedding function from seen classes and apply the function to instances from unseen classes with limited labels. This style of transfer learning is task-agnostic: the embedding function is not learned optimally discriminative with respect to the unseen classes, where discerning among them leads to the target task. In this paper, we propose a novel approach to adapt the instance embeddings to the target classification task with a set-to-set function, yielding embeddings that are task-specific and are discriminative. We empirically investigated various instantiations of such set-to-set functions and observed the Transformer is most effective -- as it naturally satisfies key properties of our desired model. We denote this model as FEAT (few-shot embedding adaptation w/ Transformer) and validate it on both the standard few-shot classification benchmark and four extended few-shot learning settings with essential use cases, i.e., cross-domain, transductive, generalized few-shot learning, and low-shot learning. It archived consistent improvements over baseline models as well as previous methods and established the new state-of-the-art results on two benchmarks.

Motivation & Objective

• Motivate the limitation of task-agnostic embeddings in few-shot learning where the unseen task requires different discriminative features.
• Propose a model-based embedding adaptation via a set-to-set function to tailor embeddings to each target task.
• Show that Transformer-based adaptation (FEAT) outperforms other instantiations and baselines across standard and extended FSL settings.
• Provide analysis on interpolation/extrapolation across tasks and demonstrate parameter efficiency of FEAT.

Proposed method

• Introduce a set-to-set transformation T that outputs adapted embeddings psi_x from the raw embeddings phi_x for all training instances in a task (permutation-invariant).
• Instantiate T with multiple architectures (Bi-LSTM, DeepSets, GCN, Transformer) and compare their task-adaptation effectiveness.
• Use nearest-neighbor classification with adapted embeddings psi_x and test embeddings phi_x for few-shot tasks.
• Apply a contrastive objective to align adapted embeddings with class centers and encourage intra-class compactness while maintaining separation from other classes.
• Pre-train backbones with an extra classification stage on seen classes and fine-tune with meta-learning over synthesized N-shot tasks.
• Empirically evaluate FEAT (Transformer-based T) against baselines on Mini/ImageNet and Tiered ImageNet, including cross-domain, transductive, and generalized FSL settings.

Experimental results

Research questions

• RQ1 Can task-specific embedding adaptation improve few-shot learning performance over task-agnostic embeddings?
• RQ2 Which set-to-set instantiation most effectively contextualizes embeddings for a given target task?
• RQ3 Does Transformer-based adaptation provide superior performance, permutation invariance, and interpolation/extrapolation capabilities compared to others?
• RQ4 How does FEAT perform in extended FSL settings such as cross-domain generalization, transductive FSL, and generalized/FSL?
• RQ5 Is the adaptation mechanism parameter-efficient while delivering strong empirical gains?

Key findings

• FEAT (Transformer-based set-to-set adaptation) consistently improves over ProtoNet and other embedding-adaptation variants across standard FSL benchmarks.
• FEAT achieves state-of-the-art results on Mini ImageNet and shows strong, consistent gains on Tiered ImageNet under ResNet backbones.
• Among instantiations, BiLSTM often underperforms due to lack of permutation invariance, while DeepSets and GCN can help but are less consistently effective than FEAT.
• FEAT demonstrates strong interpolation and extrapolation abilities across different numbers of classes (ways) and maintains high parameter efficiency.
• In extended tasks, FEAT outperforms baselines in few-shot domain generalization, transductive FSL, and generalized FSL setups.