[Paper Review] FMix: Enhancing Mixed Sample Data Augmentation

Ethan Harris, Antonia Marcu+4 more|arXiv (Cornell University)|Feb 27, 2020

Domain Adaptation and Few-Shot Learning62 references104 citations

TL;DR

FMix introduces a masking mixed-sample data augmentation using low-frequency Fourier-based masks, outperforming MixUp and CutMix across multiple datasets and modalities.

ABSTRACT

Mixed Sample Data Augmentation (MSDA) has received increasing attention in recent years, with many successful variants such as MixUp and CutMix. By studying the mutual information between the function learned by a VAE on the original data and on the augmented data we show that MixUp distorts learned functions in a way that CutMix does not. We further demonstrate this by showing that MixUp acts as a form of adversarial training, increasing robustness to attacks such as Deep Fool and Uniform Noise which produce examples similar to those generated by MixUp. We argue that this distortion prevents models from learning about sample specific features in the data, aiding generalisation performance. In contrast, we suggest that CutMix works more like a traditional augmentation, improving performance by preventing memorisation without distorting the data distribution. However, we argue that an MSDA which builds on CutMix to include masks of arbitrary shape, rather than just square, could further prevent memorisation whilst preserving the data distribution in the same way. To this end, we propose FMix, an MSDA that uses random binary masks obtained by applying a threshold to low frequency images sampled from Fourier space. These random masks can take on a wide range of shapes and can be generated for use with one, two, and three dimensional data. FMix improves performance over MixUp and CutMix, without an increase in training time, for a number of models across a range of data sets and problem settings, obtaining a new single model state-of-the-art result on CIFAR-10 without external data. Finally, we show that a consequence of the difference between interpolating MSDA such as MixUp and masking MSDA such as FMix is that the two can be combined to improve performance even further. Code for all experiments is provided at https://github.com/ecs-vlc/FMix .

Motivation & Objective

• Investigate how MSDA distortions affect learned representations and generalisation.
• Compare interpolative MSDA (MixUp) with masking MSDA (CutMix) using information-theoretic and robustness analyses.
• Propose FMix, a flexible masking MSDA with diverse mask shapes to better preserve data distribution.
• Demonstrate FMix’s effectiveness across image, audio, and 3D point cloud tasks.

Proposed method

• Define a mutual information based measure using VAEs to compare representations learned from real versus augmented data.
• Show that MixUp distorts learned functions and acts like adversarial training, while CutMix preserves more data information.
• Introduce FMix by generating binary masks from low-frequency Fourier-space samples and thresholding to create diverse, locally consistent masks.
• The FMix masking function is x_A = M ⊙ x_1 + (1−M) ⊙ x_2 with M obtained from thresholded low-frequency images.
• Evaluate FMix against baselines on CIFAR-10/100, Fashion MNIST, Tiny-ImageNet, ImageNet and additional modalities (speech, grapheme, and 3D point clouds).

Experimental results

Research questions

• RQ1 Does masking MSDA preserve data distribution better than interpolative MSDA in CNN representations?
• RQ2 Can Fourier-based random masks provide a larger, more varied augmentation space than square masks like CutMix?
• RQ3 How does FMix perform across diverse data modalities (image, audio, 3D) compared to MixUp and CutMix?
• RQ4 Do masking MSDAs complement interpolative MSDAs when combined in training policies?

Key findings

• FMix yields higher mutual information between augmented and real data representations than MixUp and CutMix in VAE-based analyses.
• FMix produces augmented data that better preserves the data distribution for CNNs, as evidenced by Grad-CAM analyses suggesting broader feature utilization.
• FMix improves classification accuracy over baselines and several MSDA methods on CIFAR-10/100, Fashion MNIST, Tiny-ImageNet, and other settings, achieving strong or state-of-the-art results without external data (e.g., CIFAR-10 with PyramidNet).
• FMix extends to one- and three-dimensional data and to other modalities (speech, grapheme, and 3D point clouds), often outperforming MixUp and CutMix.
• A hybrid policy alternating between MixUp and FMix can outperform either method alone when training data is limited.