[Paper Review] A Group-Theoretic Framework for Data Augmentation

Shuxiao Chen, Edgar Dobriban+1 more|arXiv (Cornell University)|Jul 25, 2019

Machine Learning and Algorithms95 references76 citations

TL;DR

The paper presents a group-theoretic framework showing data augmentation as averaging over group orbits, leading to variance reduction and improved sample efficiency in ERM and MLE settings, with theory, examples, and bias-variance tradeoffs for approximate invariance.

ABSTRACT

Data augmentation is a widely used trick when training deep neural networks: in addition to the original data, properly transformed data are also added to the training set. However, to the best of our knowledge, a clear mathematical framework to explain the performance benefits of data augmentation is not available. In this paper, we develop such a theoretical framework. We show data augmentation is equivalent to an averaging operation over the orbits of a certain group that keeps the data distribution approximately invariant. We prove that it leads to variance reduction. We study empirical risk minimization, and the examples of exponential families, linear regression, and certain two-layer neural networks. We also discuss how data augmentation could be used in problems with symmetry where other approaches are prevalent, such as in cryo-electron microscopy (cryo-EM).

Motivation & Objective

• Motivate and formalize data augmentation within a group-invariance framework.
• Characterize when augmentation reduces variance and improves sample efficiency in ERM and MLE.
• Develop non-asymptotic and asymptotic results linking augmentation to variance, Rademacher complexity, and Fisher information.
• Provide concrete examples (exponential families, linear regression, two-layer nets) and discuss approximate invariance.
• Suggest applications beyond deep learning to problems with symmetry (e.g., cryo-EM).

Proposed method

• Model data invariance via a group G acting on the data with X ≈d gX for g in G.
• Show that data augmentation corresponds to minimizing an augmented loss: average of the original loss over the group action.
• Introduce augmented ERM/MLE, constrained MLE, augmented MLE, invariant representations, and marginal MLE variants.
• Prove variance reduction under exact invariance through orbit averaging (Rao-Blackwellization).
• Derive non-asymptotic results: loss-averaging reduces Rademacher complexity; gradient-averaging reduces gradient variance under strong convexity.
• Provide asymptotic analysis: variance reduction depends on gradient covariances along group orbits and potential Fisher information gains.
• Extend results to approximate invariance using optimal transport to discuss bias-variance tradeoffs.
• Offer multiple examples and discuss connections to sufficiency, invariance, and regularization.

Experimental results

Research questions

• RQ1 How can data augmentation be understood as an averaging operation over a symmetry group?
• RQ2 Under exact vs approximate invariance, when does augmentation reduce variance and improve statistical efficiency?
• RQ3 How does data augmentation affect ERM and MLE in non-asymptotic and asymptotic regimes?
• RQ4 What are practical variants (constrained, augmented, invariant, marginal MLE) and their tradeoffs?
• RQ5 How can the framework be applied to problems with symmetry beyond deep learning (e.g., cryo-EM)?

Key findings

• Orbit averaging under exact invariance reduces the variance of any function.
• Loss averaging lowers the Rademacher complexity of the loss class, suggesting better generalization.
• Gradient averaging reduces the variance of the ERM when the loss is strongly convex.
• Asymptotically, variance reduction depends on the covariance of losses along the group orbit and can improve Fisher information.
• Under approximate invariance, a bias-variance tradeoff emerges governed by the orbit variability and Wasserstein distance to the transformed data.