[Paper Review] Meta-Learning with Latent Embedding Optimization

Andrei A. Rusu, Dushyant Rao+5 more|arXiv (Cornell University)|Jul 16, 2018

Domain Adaptation and Few-Shot LearningComputer Science45 references210 citations

TL;DR

LEO learns a data-dependent latent embedding of model parameters and performs gradient-based adaptation in this low-dimensional space, achieving state-of-the-art results on mini ImageNet and tiered ImageNet few-shot classification.

ABSTRACT

Gradient-based meta-learning techniques are both widely applicable and proficient at solving challenging few-shot learning and fast adaptation problems. However, they have practical difficulties when operating on high-dimensional parameter spaces in extreme low-data regimes. We show that it is possible to bypass these limitations by learning a data-dependent latent generative representation of model parameters, and performing gradient-based meta-learning in this low-dimensional latent space. The resulting approach, latent embedding optimization (LEO), decouples the gradient-based adaptation procedure from the underlying high-dimensional space of model parameters. Our evaluation shows that LEO can achieve state-of-the-art performance on the competitive miniImageNet and tieredImageNet few-shot classification tasks. Further analysis indicates LEO is able to capture uncertainty in the data, and can perform adaptation more effectively by optimizing in latent space.

Motivation & Objective

• Address the sample efficiency gap in gradient-based meta-learning for high-dimensional models under extreme low-data regimes.
• Propose a low-dimensional latent space to generate and adapt model parameters conditioned on few-shot data.
• Demonstrate state-of-the-art performance on mini ImageNet and tiered ImageNet few-shot tasks.
• Investigate whether latent-space optimization captures uncertainty and supports multimodal task distributions.

Proposed method

• Introduce Latent Embedding Optimization (LEO) that learns a stochastic latent space conditioned on few-shot data.
• Encode few-shot samples into latent codes using a relation network to capture class relationships.
• Decode latent codes to generate task-specific top-layer parameters of a classifier.
• Perform inner-loop gradient updates with respect to latent codes, then decode updated codes to new parameters.
• Meta-train by backpropagating validation loss through the inner loop and through the encoder/decoder networks with a KL-regularized latent space.
• Utilize a data-conditioned encoder and a stochastic, low-dimensional latent space to enable gradient-based adaptation in parameter space via a differentiable decoder.

Experimental results

Research questions

• RQ1 Can a low-dimensional latent space over model parameters, conditioned on few-shot data, support effective gradient-based adaptation?
• RQ2 Does optimizing in latent space improve few-shot generalization and capture uncertainty in multimodal task distributions?
• RQ3 How does LEO compare to existing optimization-based meta-learning methods on standard few-shot benchmarks?
• RQ4 Is a data-dependent parameter initialization via latent codes critical for performance gains?

Key findings

• LEO achieves state-of-the-art 1-shot and 5-shot accuracy on mini ImageNet and tiered ImageNet scales with both shallow and deep backbones.
• On mini ImageNet, LEO attains 61.76% (1-shot) and 77.59% (5-shot) in the reported setting, outperforming prior methods.
• On tiered ImageNet, LEO achieves 66.33% (1-shot) and 81.44% (5-shot), surpassing previous state-of-the-art.
• Ablation shows that both the data-conditioned latent encoding and latent space adaptation are critical for performance, with latent bottleneck greatly enhancing generalization over direct parameter-space adaptation.
• Latent embeddings visualize substantial adaptation, indicating the model moves from encoder-derived codes to task-specific adapted codes under LEO.