[論文レビュー] A Group-Theoretic Framework for Data Augmentation

Shuxiao Chen, Edgar Dobriban+1 more|arXiv (Cornell University)|Jul 25, 2019

Machine Learning and Algorithms参考文献 95被引用数 76

ひとこと要約

この論文は、データ拡張をグループ軌道の平均化として表す群論的枠組みを提示し、ERMとMLE設定における分散削減とサンプル効率の改善をもたらすことを示す。理論、例、近似不変性に対するバイアス-分散トレードオフを含む。

ABSTRACT

Data augmentation is a widely used trick when training deep neural networks: in addition to the original data, properly transformed data are also added to the training set. However, to the best of our knowledge, a clear mathematical framework to explain the performance benefits of data augmentation is not available. In this paper, we develop such a theoretical framework. We show data augmentation is equivalent to an averaging operation over the orbits of a certain group that keeps the data distribution approximately invariant. We prove that it leads to variance reduction. We study empirical risk minimization, and the examples of exponential families, linear regression, and certain two-layer neural networks. We also discuss how data augmentation could be used in problems with symmetry where other approaches are prevalent, such as in cryo-electron microscopy (cryo-EM).

研究の動機と目的

• Motivate and formalize data augmentation within a group-invariance framework.
• Characterize when augmentation reduces variance and improves sample efficiency in ERM and MLE.
• Develop non-asymptotic and asymptotic results linking augmentation to variance, Rademacher complexity, and Fisher information.
• Provide concrete examples (exponential families, linear regression, two-layer nets) and discuss approximate invariance.
• Suggest applications beyond deep learning to problems with symmetry (e.g., cryo-EM).

提案手法

• Model data invariance via a group G acting on the data with X ≈d gX for g in G.
• Show that data augmentation corresponds to minimizing an augmented loss: average of the original loss over the group action.
• Introduce augmented ERM/MLE, constrained MLE, augmented MLE, invariant representations, and marginal MLE variants.
• Prove variance reduction under exact invariance through orbit averaging (Rao-Blackwellization).
• Derive non-asymptotic results: loss-averaging reduces Rademacher complexity; gradient-averaging reduces gradient variance under strong convexity.
• Provide asymptotic analysis: variance reduction depends on the covariance of losses along group orbits and potential Fisher information gains.
• Extend results to approximate invariance using optimal transport to discuss bias-variance tradeoffs.
• Offer multiple examples and discuss connections to sufficiency, invariance, and regularization.

実験結果

リサーチクエスチョン

• RQ1 How can data augmentation be understood as an averaging operation over a symmetry group?
• RQ2 Under exact vs approximate invariance, when does augmentation reduce variance and improve statistical efficiency?
• RQ3 How does data augmentation affect ERM and MLE in non-asymptotic and asymptotic regimes?
• RQ4 What are practical variants (constrained, augmented, invariant, marginal MLE) and their tradeoffs?
• RQ5 How can the framework be applied to problems with symmetry beyond deep learning (e.g., cryo-EM)?

主な発見

• Orbit averaging under exact invariance reduces the variance of any function.
• Loss averaging lowers the Rademacher complexity of the loss class, suggesting better generalization.
• Gradient averaging reduces the variance of the ERM when the loss is strongly convex.
• Asymptotically, variance reduction depends on the covariance of losses along the group orbit and can improve Fisher information.
• Under approximate invariance, a bias-variance tradeoff emerges governed by the orbit variability and Wasserstein distance to the transformed data.