[Paper Review] Meta Dropout: Learning to Perturb Features for Generalization

Haebeom Lee, Taewook Nam+2 more|arXiv (Cornell University)|May 30, 2019

Domain Adaptation and Few-Shot Learning48 references4 citations

TL;DR

This paper proposes Meta-Dropout, a meta-learning-based regularization method that trains an input-dependent noise generator to perturb latent features during training, improving generalization in few-shot learning. By learning to adaptively perturb features based on input context, Meta-Dropout significantly outperforms existing methods like manifold mixup and information dropout on Omniglot and miniImageNet benchmarks.

ABSTRACT

A machine learning model that generalizes well should obtain low errors on unseen test examples. Thus, if we know how to optimally perturb training examples to account for test examples, we may achieve better generalization performance. However, obtaining such perturbation is not possible in standard machine learning frameworks as the distribution of the test data is unknown. To tackle this challenge, we propose a novel regularization method, meta-dropout, which learns to perturb the latent features of training examples for generalization in a meta-learning framework. Specifically, we meta-learn a noise generator which outputs a multiplicative noise distribution for latent features, to obtain low errors on the test instances in an input-dependent manner. Then, the learned noise generator can perturb the training examples of unseen tasks at the meta-test time for improved generalization. We validate our method on few-shot classification datasets, whose results show that it significantly improves the generalization performance of the base model, and largely outperforms existing regularization methods such as information bottleneck, manifold mixup, and information dropout.

Motivation & Objective

• To address the challenge of improving model generalization when test data distribution is unknown during training.
• To develop a regularization method that explicitly minimizes test loss during meta-training by learning optimal perturbations of latent features.
• To enable transferable, input-specific noise generation that adapts to unseen tasks in few-shot learning.
• To outperform existing regularization techniques such as information bottleneck, manifold mixup, and information dropout.
• To provide a probabilistic interpretation of the method as meta-learning in a variational inference framework.

Proposed method

• The method trains a noise generator using meta-learning to produce multiplicative noise distributions over latent features based on input-dependent features from lower layers.
• The noise generator is trained end-to-end via a meta-learning objective that minimizes expected test loss across multiple tasks with random train/test splits.
• The noise is applied as stochastic multiplicative perturbations to latent features during training, with the noise distribution being input-specific and adaptive.
• The framework is interpreted as meta-learning the variational inference process for a graphical model, where the noise generator acts as a posterior approximation.
• The method is applied within a MAML-based few-shot learning framework, with the noise generator trained jointly using inner and outer loop optimization.
• The noise generator uses fixed variance (e.g., identity matrix) but is conditioned on input features, enabling adaptive, task-generalizable perturbations.

Experimental results

Research questions

• RQ1 Can a meta-learned, input-dependent noise generator improve generalization in few-shot learning beyond standard regularization?
• RQ2 How does input-dependent noise compare to fixed or input-independent noise in terms of generalization performance?
• RQ3 Can meta-learned perturbations be interpreted as a form of variational inference in a graphical model?
• RQ4 Does Meta-Dropout improve robustness against adversarial attacks compared to baseline models?
• RQ5 How does the performance of Meta-Dropout compare to state-of-the-art regularizers like manifold mixup and information dropout?

Key findings

• Meta-Dropout achieves 96.63% accuracy on Omniglot 1-shot (ℓ₁) and 98.73% on ℓ₂, significantly outperforming MAML and other baselines.
• On Omniglot 5-shot (ℓ∞), Meta-Dropout reaches 67.42% accuracy, outperforming the next best method by over 4 percentage points.
• The method with input-dependent noise consistently outperforms fixed Gaussian, independent Gaussian, and weight-learned Gaussian baselines, demonstrating the importance of input-specific perturbations.
• Meta-Dropout with fixed variance outperforms a variant that learns variance adaptively, suggesting that fixed noise scale is more effective for generalization.
• The ablation study confirms that both the deterministic component (learned scaling) and stochastic component (noise injection) are essential, with the combination yielding the best performance.
• Meta-Dropout shows improved adversarial robustness, maintaining higher clean accuracy under PGD attacks compared to standard MAML and other baselines.