[Paper Review] Preventing Manifold Intrusion with Locality: Local Mixup

Raphaël Baena, Lucas Drumetz+1 more|arXiv (Cornell University)|Jan 12, 2022

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper proposes Local Mixup, a data augmentation technique that improves generalization by weighting Mixup virtual samples based on input sample proximity, reducing manifold intrusion. It achieves better test accuracy than both vanilla training and standard Mixup on CIFAR-10, Fashion-MNIST, and SVHN by enabling a learnable bias-variance trade-off through a single locality hyperparameter.

ABSTRACT

Mixup is a data-dependent regularization technique that consists in linearly interpolating input samples and associated outputs. It has been shown to improve accuracy when used to train on standard machine learning datasets. However, authors have pointed out that Mixup can produce out-of-distribution virtual samples and even contradictions in the augmented training set, potentially resulting in adversarial effects. In this paper, we introduce Local Mixup in which distant input samples are weighted down when computing the loss. In constrained settings we demonstrate that Local Mixup can create a trade-off between bias and variance, with the extreme cases reducing to vanilla training and classical Mixup. Using standardized computer vision benchmarks , we also show that Local Mixup can improve test accuracy.

Motivation & Objective

• To address the issue of manifold intrusion in Mixup, where distant samples produce out-of-distribution or contradictory virtual samples.
• To introduce a locality-aware regularization method that controls the bias-variance trade-off during training.
• To demonstrate that incorporating local structure in Mixup improves generalization on standard computer vision datasets.
• To provide a continuous interpolation between vanilla training and classical Mixup via a single hyperparameter.

Proposed method

• Local Mixup introduces a distance-dependent weighting scheme where virtual samples are weighted by a function of the Euclidean distance between input pairs.
• The method uses a smooth decreasing exponential function parameterized by α to attenuate the loss contribution of distant sample pairs.
• Virtual samples are generated via linear interpolation: x̃ = λxi + (1−λ)xj and ỹ = λyi + (1−λ)yj, but only contribute to the loss with a weight dependent on ||xi − xj||.
• The approach allows a continuous transition from vanilla training (α → 0) to standard Mixup (α → ∞), with intermediate values balancing bias and variance.
• The method is evaluated on CIFAR-10, Fashion-MNIST, and SVHN using standard architectures (ResNet18, DenseNet, LeNet-5), with hyperparameters selected for optimal test error.

Experimental results

Research questions

• RQ1 Can locality-aware weighting in Mixup reduce manifold intrusion and improve generalization?
• RQ2 How does Local Mixup affect the bias-variance trade-off in both low- and high-dimensional settings?
• RQ3 Does Local Mixup consistently outperform standard Mixup and vanilla training on standard vision benchmarks?
• RQ4 What is the impact of the locality hyperparameter α on model performance and Lipschitz continuity?

Key findings

• On CIFAR-10, Local Mixup with α = 3e−3 achieved a test error rate of 4.03% ± 0.03, outperforming vanilla ResNet18 (4.98% ± 0.03) and standard Mixup (4.13% ± 0.03).
• On Fashion-MNIST, Local Mixup with α = 1e−3 achieved 5.97% ± 0.2 error, outperforming both baseline (6.20% ± 0.2) and Mixup (6.36% ± 0.16).
• On SVHN, Local Mixup with α = 5e−2 achieved 8.20% ± 0.13 error, outperforming vanilla LeNet (10.01% ± 0.15) and Mixup (8.31% ± 0.14).
• The method provides a continuous interpolation between vanilla training and classical Mixup, with the extreme cases corresponding to no augmentation and full Mixup, respectively.
• Experiments show that Local Mixup can tune a lower bound on the Lipschitz constant of the trained model, indicating improved robustness.
• Alternative graphs (K-nearest neighbor or thresholded) did not outperform Local Mixup, suggesting that the exponential decay weighting is effective for locality control.