[Paper Review] Implicit Semantic Data Augmentation for Deep Networks

Yulin Wang, Xuran Pan+4 more|arXiv (Cornell University)|Sep 26, 2019

Multimodal Machine Learning ApplicationsComputer Science60 citations

TL;DR

ISDA implicitly augments training by perturbing deep features along class-conditional semantic directions, using an online covariance estimate to derive a robust cross-entropy loss that improves generalization without extra data generation.

ABSTRACT

In this paper, we propose a novel implicit semantic data augmentation (ISDA) approach to complement traditional augmentation techniques like flipping, translation or rotation. Our work is motivated by the intriguing property that deep networks are surprisingly good at linearizing features, such that certain directions in the deep feature space correspond to meaningful semantic transformations, e.g., adding sunglasses or changing backgrounds. As a consequence, translating training samples along many semantic directions in the feature space can effectively augment the dataset to improve generalization. To implement this idea effectively and efficiently, we first perform an online estimate of the covariance matrix of deep features for each class, which captures the intra-class semantic variations. Then random vectors are drawn from a zero-mean normal distribution with the estimated covariance to augment the training data in that class. Importantly, instead of augmenting the samples explicitly, we can directly minimize an upper bound of the expected cross-entropy (CE) loss on the augmented training set, leading to a highly efficient algorithm. In fact, we show that the proposed ISDA amounts to minimizing a novel robust CE loss, which adds negligible extra computational cost to a normal training procedure. Although being simple, ISDA consistently improves the generalization performance of popular deep models (ResNets and DenseNets) on a variety of datasets, e.g., CIFAR-10, CIFAR-100 and ImageNet. Code for reproducing our results is available at https://github.com/blackfeather-wang/ISDA-for-Deep-Networks.

Motivation & Objective

• Motivate semantic transformations beyond standard augmentations to improve generalization in image classification.
• Propose an efficient implicit augmentation mechanism that operates in feature space rather than generating explicit augmented samples.
• Develop a closed-form upper bound loss that enables scalable optimization with existing architectures.
• Demonstrate that ISDA consistently improves performance on CIFAR-10/100 and ImageNet across multiple architectures.

Proposed method

• Estimate class-conditional feature covariance online during training.
• Sample random directions from N(0, lambda * Sigma_y) in feature space and conceptually translate features along these directions.
• Derive a closed-form upper bound on the expected cross-entropy loss under augmentation, producing a robust loss ar{L}_infty.
• Optimize the robust loss with SGD without explicitly generating augmented samples.
• Provide Algorithm 1 detailing ISDA steps and outline covariance estimation in supplementary material.

Experimental results

Research questions

• RQ1 Can implicit semantic directions in deep feature space be leveraged to meaningfully augment training data without explicit sample generation?
• RQ2 Does class-conditional covariance capture intra-class semantic variations that improve generalization across datasets and architectures?
• RQ3 How does ISDA interact with non-semantic augmentations like Cutout or AutoAugment?
• RQ4 Is the ISDA loss practically implementable with negligible computational overhead on large-scale datasets?

Key findings

• ISDA yields consistent generalization gains across ResNet, DenseNet, and related architectures on CIFAR-10, CIFAR-100, and ImageNet.
• ISDA provides notable improvements when combined with non-semantic augmentations such as Cutout and AutoAugment.
• The method can be implemented as a robust loss function with negligible extra computation, and reduces to standard cross-entropy as lambda approaches zero.
• ISDA often outperforms state-of-the-art robust losses and GAN-based semantic augmentation methods in reported experiments.
• Ablation studies show that using full class-conditioned covariance (vs. identity/diagonal or a single global covariance) is important for effectiveness.