[Paper Review] OpenMixup: Open Mixup Toolbox and Benchmark for Visual Representation Learning

Siyuan Li, Zedong Wang+6 more|arXiv (Cornell University)|Sep 11, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

OpenMixup introduces the first comprehensive benchmark and unified open-source framework for visual representation learning using mixup data augmentation. It enables standardized, large-scale evaluation of 12 diverse datasets, revealing that dynamic mixup methods (e.g., AutoMix, SAMix) offer superior robustness, training stability, and transferability compared to static methods, while highlighting trade-offs in compatibility and efficiency across backbones and architectures.

ABSTRACT

Mixup augmentation has emerged as a widely used technique for improving the generalization ability of deep neural networks (DNNs). However, the lack of standardized implementations and benchmarks has impeded recent progress, resulting in poor reproducibility, unfair comparisons, and conflicting insights. In this paper, we introduce OpenMixup, the first mixup augmentation codebase, and benchmark for visual representation learning. Specifically, we train 18 representative mixup baselines from scratch and rigorously evaluate them across 11 image datasets of varying scales and granularity, ranging from fine-grained scenarios to complex non-iconic scenes. We also open-source our modular codebase, including a collection of popular vision backbones, optimization strategies, and analysis toolkits, which not only supports the benchmarking but enables broader mixup applications beyond classification, such as self-supervised learning and regression tasks. Through experiments and empirical analysis, we gain observations and insights on mixup performance-efficiency trade-offs, generalization, and optimization behaviors, and thereby identify preferred choices for different needs. To the best of our knowledge, OpenMixup has facilitated several recent studies. We believe this work can further advance reproducible mixup augmentation research and thereby lay a solid ground for future progress in the community. The source code and user documents are available at \url{https://github.com/Westlake-AI/openmixup}.

Motivation & Objective

• To address the lack of a standardized, comprehensive benchmark for evaluating mixup methods in visual representation learning.
• To provide a unified, modular, and extensible codebase for consistent implementation, training, and evaluation of diverse mixup techniques.
• To conduct large-scale, fair comparisons across 12 diverse image datasets, controlling for confounders like backbone choice and data distribution.
• To empirically analyze how mixup policies, network architectures, and dataset properties influence model performance and generalization.
• To offer actionable insights for future development of more robust, efficient, and transferable mixup-based methods.

Proposed method

• The OpenMixup framework integrates a wide range of mixup algorithms—spanning static (e.g., Mixup, CutMix) and dynamic (e.g., AutoMix, SAMix, Co-Mixup) methods—into a single modular training pipeline.
• It supports multiple backbones (CNNs, ViTs, efficient networks) and modules, enabling systematic ablation across model architectures.
• The framework enables controlled evaluation with confounder tweaking (e.g., mixing ratio, data split, augmentation schedule) to ensure fair comparisons.
• It includes model analysis toolkits such as class activation mapping (CAM) visualization for assessing localizability and downstream transferability.
• The benchmark is evaluated on 12 diverse image classification datasets, including ImageNet-1K, Places205, and FGVC-Aircraft, under standardized training protocols.
• The framework is open-sourced and extensible, supporting future integration of new mixup methods and downstream tasks like object detection.

Experimental results

Research questions

• RQ1 How do different mixup policies (static vs. dynamic) compare in terms of accuracy, robustness, and training stability across diverse datasets?
• RQ2 How does backbone architecture selection affect the performance and generalization of mixup methods?
• RQ3 To what extent do mixup methods transfer to downstream detection tasks, and which exhibit better localizability?
• RQ4 What are the trade-offs between computational efficiency, compatibility, and performance across mixup methods?
• RQ5 How do dataset-specific properties (e.g., class imbalance, image complexity) influence the effectiveness of mixup strategies?

Key findings

• Dynamic mixup methods (e.g., AutoMix, SAMix) consistently outperform static mixup across most datasets, showing superior robustness to backbone selection and better training stability.
• AutoMix and SAMix exhibit exceptional robustness and convergence, with higher top-1 accuracy and more stable training curves than most static methods.
• Static Mixup (Zhang et al., 2018) shows superior training stability despite lower accuracy, likely due to its convex interpolation nature.
• CutMix (Yun et al., 2019) demonstrates strong compatibility with both modern CNNs (e.g., ConvNeXt, ResNeXt) and ViTs (e.g., DeiT), indicating broad applicability.
• ViT-specific dynamic methods (e.g., TransMix, TokenMix) perform exceptionally well on small ViTs (DeiT-S) but degrade significantly on larger or smaller variants (e.g., PVT-T), indicating high sensitivity to model scale.
• AutoMix and SAMix show superior localizability via CAM visualization, suggesting hierarchical advantages for detection transfer, despite higher computational cost.