[Paper Review] Manifold Mixup: Better Representations by Interpolating Hidden States

Vikas Verma, Alex Lamb+6 more|arXiv (Cornell University)|Jun 13, 2018

Generative Adversarial Networks and Image Synthesis477 citations

TL;DR

Manifold Mixup regularizes networks by interpolating hidden representations (with mixed labels), yielding flatter class representations, smoother decision boundaries, better generalization, and improved robustness to novel deformations and single-step adversarial attacks.

ABSTRACT

Deep neural networks excel at learning the training data, but often provide incorrect and confident predictions when evaluated on slightly different test examples. This includes distribution shifts, outliers, and adversarial examples. To address these issues, we propose Manifold Mixup, a simple regularizer that encourages neural networks to predict less confidently on interpolations of hidden representations. Manifold Mixup leverages semantic interpolations as additional training signal, obtaining neural networks with smoother decision boundaries at multiple levels of representation. As a result, neural networks trained with Manifold Mixup learn class-representations with fewer directions of variance. We prove theory on why this flattening happens under ideal conditions, validate it on practical situations, and connect it to previous works on information theory and generalization. In spite of incurring no significant computation and being implemented in a few lines of code, Manifold Mixup improves strong baselines in supervised learning, robustness to single-step adversarial attacks, and test log-likelihood.

Motivation & Objective

• Motivate the need for better generalization under distribution shifts and adversarial perturbations in deep nets.
• Introduce Manifold Mixup as a regularizer that performs linear interpolations in hidden representations with corresponding soft labels.
• Theoretically characterize representation flattening and empirically validate its occurrence and benefits across datasets.
• Demonstrate empirical gains in generalization, log-likelihood, and robustness to adversarial and novel deformations.

Proposed method

• Train networks with Manifold Mixup by selecting a random layer k from a set of eligible layers, computing Mixλ on hidden representations gk(x) and gk(x′) and their one-hot labels, and backpropagating through the full network.
• Minimize the loss averaged over pairs of examples and Mixλ, i.e., L(f)=E_{(x,y),(x′,y′),λ,k} ℓ(fk(Mixλ(gk(x),gk(x′))), Mixλ(y,y′)).
• Compare Manifold Mixup to Input Mixup and other regularizers across benchmarks (CIFAR-10/100, SVHN, TinyImagenet) and analyze representations via SVD.
• Provide theoretical results showing flattening: if hidden layer dimension dim(H) ≥ d−1 (d = number of classes), the optimal f⋆ is linear and training points lie on a dim(H)−d+1 subspace (Theorem 1 and Corollary 1).
• Examine robustness to deformations and adversarial attacks (FGSM, PGD) and analyze gradient behavior to ensure gradient signal integrity (anti-gradient masking checks).
• Demonstrate that Manifold Mixup yields better NLL and test error across architectures (PreActResNet variants, Wide-ResNet) and datasets.

Experimental results

Research questions

• RQ1 Does interpolating hidden representations with Mixλ improve generalization and calibration (NLL) compared to input-space mixup and other regularizers?
• RQ2 How does Manifold Mixup affect the geometry of class representations in hidden layers (flattening, reduced variance directions)?
• RQ3 Does the method enhance robustness to novel input deformations and single-step adversarial attacks, and to what extent?
• RQ4 Is the improvement due to training with interpolations in deep representations or merely the regularization signal at the input layer?
• RQ5 How sensitive is the method to hyperparameters like α and the set of eligible layers S?

Key findings

• Manifold Mixup achieves better test error and significantly lower NLL than vanilla training, AdaMix, and Input Mixup across CIFAR-10/100, SVHN, and TinyImagenet.
• It provides improved robustness to novel deformations, with higher accuracy on rotated, sheared, and zoomed inputs compared to baselines.
• FGSM adversarial accuracy improves notably with Manifold Mixup, though PGD robustness does not show major gains.
• SVD analyses show Manifold Mixup flattens class-specific representations, reducing the sum of the smaller singular values relative to the largest, indicating lower dimensional dispersion than other regularizers.
• Theoretical results show that with sufficiently high hidden dimensionality, mixing in hidden layers can drive the loss to zero with linear decision boundaries, implying flattening of class representations.