[Paper Review] Synergy and Symmetry in Deep Learning: Interactions between the Data, Model, and Inference Algorithm

Lechao Xiao, Jeffrey Pennington|arXiv (Cornell University)|Jul 11, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper identifies that the synergy between data, model, and inference algorithm—particularly through symmetry compatibility—enables deep learning to overcome the curse of dimensionality. It demonstrates that learning efficiency improves with more data when symmetries in data, model, and optimization algorithm are aligned, revealing a data-improving-data-efficiency (DIDE) effect where scaling data enhances model generalization.

ABSTRACT

Although learning in high dimensions is commonly believed to suffer from the curse of dimensionality, modern machine learning methods often exhibit an astonishing power to tackle a wide range of challenging real-world learning problems without using abundant amounts of data. How exactly these methods break this curse remains a fundamental open question in the theory of deep learning. While previous efforts have investigated this question by studying the data (D), model (M), and inference algorithm (I) as independent modules, in this paper, we analyze the triplet (D, M, I) as an integrated system and identify important synergies that help mitigate the curse of dimensionality. We first study the basic symmetries associated with various learning algorithms (M, I), focusing on four prototypical architectures in deep learning: fully-connected networks (FCN), locally-connected networks (LCN), and convolutional networks with and without pooling (GAP/VEC). We find that learning is most efficient when these symmetries are compatible with those of the data distribution and that performance significantly deteriorates when any member of the (D, M, I) triplet is inconsistent or suboptimal.

Motivation & Objective

• To understand how modern deep learning systems break the curse of dimensionality despite high input dimensionality.
• To investigate the role of symmetries in data, model architecture, and inference algorithms as a unifying principle for learning efficiency.
• To analyze the interplay between data distribution, model inductive biases, and optimization dynamics in mitigating sample complexity.
• To identify conditions under which learning efficiency improves with more data, challenging classical statistical learning bounds.

Proposed method

• Analyzes four prototypical deep learning architectures: fully-connected networks (FCN), locally-connected networks (LCN), and convolutional networks with vectorized (VEC) and global average pooled (GAP) readouts.
• Uses the infinite-width limit to study symmetries of models and inference algorithms, enabling tractable theoretical analysis of inductive biases.
• Introduces a symmetry-based framework to assess consistency between data distribution symmetries (e.g., O(3)^d rotational symmetry), model symmetries, and optimization-induced inductive biases.
• Empirically evaluates learning curves under symmetry-breaking perturbations (e.g., rotating ImageNet inputs) to measure data efficiency and detect transitions in function class.
• Quantifies the DIDE (Data Improves Data Efficiency) effect by measuring changes in scaling law slopes (α) as training data increases.
• Uses dimension counting to estimate the number of training samples needed to overcome spurious symmetries, such as O(3)^d or O(3d).

Experimental results

Research questions

• RQ1 How do symmetries in data, model, and inference algorithm jointly influence learning efficiency in high-dimensional deep learning?
• RQ2 Under what conditions does increasing training data improve data efficiency, and what causes this improvement?
• RQ3 What is the role of symmetry compatibility between data distribution and model architecture in generalization?
• RQ4 Can deep learning systems overcome spurious symmetries like O(3)^d, and if so, how many samples are required to achieve this?
• RQ5 How does the implicit bias of SGD interact with model and data symmetries to shape generalization performance?

Key findings

• The optimal learning system (ResNet101 on ImageNet with SGD) exhibits a power-law scaling of α ≈ 0.41, indicating high data efficiency.
• When data symmetry is broken (e.g., ImageNet rotated by O(3)^d), learning curves show a cusp at ~2×10⁵ samples, with a significant improvement in scaling slope (α ≈ 0.49) after the cusp.
• Models like ResNet101 and Mixer achieve α ≈ 0.49 and 0.38 in the second phase after the cusp, approaching the performance of the optimal baseline.
• The ResNet family reduces top-1 accuracy gap from ~10% (ResNet-18) to ~6% (ResNet-200) when training on O(3)^d-rotated ImageNet, showing improved robustness to symmetry-breaking.
• The EfficientNet family reduces the accuracy gap from ~4% (B0) to ~1% (B7) under the same rotation, demonstrating that wider and deeper models better overcome spurious symmetries.
• The number of samples needed to overcome O(3)^d symmetry is estimated to be ~10¹⁰, consistent with dimension counting of the symmetry group’s dimension.