[Paper Review] OnlineAugment: Online Data Augmentation with Less Domain Knowledge

Zhiqiang Tang, Yunhe Gao+4 more|ArXiv.org|Jul 17, 2020

Domain Adaptation and Few-Shot Learning43 references4 citations

TL;DR

OnlineAugment proposes an online, meta-learned data augmentation framework that co-trains three learnable networks—A-STN, D-VAE, and P-VAE—to generate adaptive, domain-agnostic augmentations during training. Unlike offline methods, it dynamically adjusts augmentations based on model state, achieving performance on par with AutoAugment and further boosting state-of-the-art results when combined with offline policies.

ABSTRACT

Data augmentation is one of the most important tools in training modern deep neural networks. Recently, great advances have been made in searching for optimal augmentation policies in the image classification domain. However, two key points related to data augmentation remain uncovered by the current methods. First is that most if not all modern augmentation search methods are offline and learning policies are isolated from their usage. The learned policies are mostly constant throughout the training process and are not adapted to the current training model state. Second, the policies rely on class-preserving image processing functions. Hence applying current offline methods to new tasks may require domain knowledge to specify such kind of operations. In this work, we offer an orthogonal online data augmentation scheme together with three new augmentation networks, co-trained with the target learning task. It is both more efficient, in the sense that it does not require expensive offline training when entering a new domain, and more adaptive as it adapts to the learner state. Our augmentation networks require less domain knowledge and are easily applicable to new tasks. Extensive experiments demonstrate that the proposed scheme alone performs on par with the state-of-the-art offline data augmentation methods, as well as improving upon the state-of-the-art in combination with those methods. Code is available at https://github.com/zhiqiangdon/online-augment .

Motivation & Objective

• Address the limitations of offline data augmentation, which rely on fixed, hand-crafted policies and require expensive pre-training or search.
• Reduce dependency on domain-specific knowledge by replacing traditional image processing operations with learnable neural networks for augmentation.
• Enable online adaptation of data augmentation to the current state of the learning model throughout training, improving generalization and efficiency.
• Develop a unified framework that is orthogonal to existing offline methods, allowing for joint use to further boost performance.
• Demonstrate generalization beyond image classification by applying the method to medical image segmentation, showing improved robustness on challenging datasets.

Proposed method

• Introduce three learnable augmentation networks: A-STN (affine spatial transformer network), D-VAE (deformation variational autoencoder), and P-VAE (perturbation variational autoencoder), which generate diverse augmentations without relying on predefined image operations.
• Co-train the augmentation networks and the target model end-to-end using a meta-learning objective that encourages generalization across training steps.
• Apply adversarial training to the augmentation networks to prevent overfitting and improve robustness of the generated augmentations.
• Use task-specific regularization losses to guide the augmentation networks in learning meaningful, class-preserving transformations.
• Integrate both meta-learning and adversarial training to enable the augmentation networks to adaptively adjust their transformation magnitude based on the current model state.
• Train the augmentation networks in real time during the main training loop, eliminating the need for separate offline policy search or pre-training.

Experimental results

Research questions

• RQ1 Can online, adaptive data augmentation outperform or complement state-of-the-art offline augmentation policies like AutoAugment in image classification tasks?
• RQ2 To what extent can learnable augmentation networks reduce the need for domain-specific knowledge in designing data augmentation operations?
• RQ3 How does the integration of meta-learning and adversarial training improve the generalization and robustness of online data augmentation?
• RQ4 Can online data augmentation be effectively transferred to non-classification tasks, such as medical image segmentation?
• RQ5 Does the dynamic adaptation of augmentation magnitude during training lead to better model convergence and performance?

Key findings

• OnlineAugment alone achieves test error rates comparable to AutoAugment on CIFAR-10 (2.4% error), CIFAR-100 (16.6%), and ImageNet (22.5%), demonstrating strong standalone performance.
• When combined with AutoAugment, OnlineAugment reduces test error to 2.0% on CIFAR-10, 16.3% on CIFAR-100, and 22.0% on ImageNet, setting new state-of-the-art results.
• On the LiTS liver and tumor segmentation dataset, OnlineAugment achieves a 73.35% average Dice score—significantly outperforming RandAugment (71.51%) when combining spatial and deformation augmentations.
• The A-STN and D-VAE modules generate diverse, adaptive augmentations that evolve during training, with augmentation magnitude decreasing as the model converges, indicating dynamic adaptation.
• The method improves performance on tumor segmentation more than on liver segmentation, suggesting it generates harder, more informative samples that benefit from adaptive augmentation.
• The combination of A-STN and D-VAE yields the best performance, demonstrating that the proposed modules are complementary and generalize well across different augmentation types.