[Paper Review] Low-Shot Learning from Imaginary Data

Yu-Xiong Wang, Ross Girshick+2 more|arXiv (Cornell University)|Jan 16, 2018

Domain Adaptation and Few-Shot LearningComputer Science34 references70 citations

TL;DR

A meta-learning framework-Augmented by a learned hallucinator that generates useful synthetic examples to improve low-shot (few-shot) classification; achieves state-of-the-art on ImageNet low-shot benchmarks, with gains up to 6 percentage points for single-example novel classes.

ABSTRACT

Humans can quickly learn new visual concepts, perhaps because they can easily visualize or imagine what novel objects look like from different views. Incorporating this ability to hallucinate novel instances of new concepts might help machine vision systems perform better low-shot learning, i.e., learning concepts from few examples. We present a novel approach to low-shot learning that uses this idea. Our approach builds on recent progress in meta-learning ("learning to learn") by combining a meta-learner with a "hallucinator" that produces additional training examples, and optimizing both models jointly. Our hallucinator can be incorporated into a variety of meta-learners and provides significant gains: up to a 6 point boost in classification accuracy when only a single training example is available, yielding state-of-the-art performance on the challenging ImageNet low-shot classification benchmark.

Motivation & Objective

• Motivate and address low-shot learning by leveraging human-like imagination to generate useful training variations.
• Propose a unified meta-learning framework that jointly learns a classifier and a hallucinator to produce informative synthetic examples.
• Demonstrate that learned hallucinations improve novel-class accuracy across multiple meta-learning architectures.
• Evaluate on a realistic ImageNet-based low-shot benchmark to establish state-of-the-art performance.

Proposed method

• Define a hallucinator G that maps a seed real example x and a noise vector z to a hallucinated example x' = G(x, z; w_G).
• Jointly train the hallucinator G and the meta-learner h through end-to-end optimization using a differentiable learning objective.
• During meta-testing, augment the training set with hallucinated examples and train h on the augmented set; during meta-training, backpropagate through h and G to optimize w_G and meta-learner parameters.
• Show that hallucination need not be realistic or perfectly diverse; instead, it should produce samples that are informative for the classifier, learned via meta-learning.
• Apply the approach to prototypical networks, matching networks, and the proposed prototype matching networks, illustrating generality across meta-learning methods.
• Describe hallucinator initialization and architecture (three-layer MLP with ReLU; block-diagonal identity initialization) and that training uses differentiable classifiers like PN, MN, PMN.

Experimental results

Research questions

• RQ1 Can a learned hallucinator produce synthetic examples that meaningfully improve low-shot classification performance when used with meta-learning?
• RQ2 Is the improvement from learned hallucination robust across different meta-learning architectures (e.g., PN, MN, PMN)?
• RQ3 Does joint optimization of the hallucinator and classifier outperform baselines using handcrafted or noise-based augmentation?
• RQ4 How does hallucination affect performance on novel classes versus base classes in a joint base+novel evaluation setup, and how does priors over novel classes influence results?

Key findings

• Hallucination trained end-to-end with meta-learning yields significant gains in novel-class accuracy, up to about 6 percentage points for prototypical networks when only one example is available.
• G produces improvements across multiple meta-learning architectures (PN, MN, PMN), especially for the novel classes, with diminished gains as more novel examples are provided but remaining significant up to certain n values.
• The learned hallucinator enhances robustness to mis-calibration and improves performance in joint base+novel evaluations, particularly when a prior over novel classes is not perfectly tuned.
• Compared to prior heuristics-based hallucination, the learned approach substantially outperforms baselines, establishing state-of-the-art results on the ImageNet low-shot benchmark.
• Ablation studies show that simple jittering or untrained Gaussian hallucinators underperform compared to the learned, meta-trained hallucinator, underscoring the need for task-specific learning of hallucinations.