[Paper Review] WHY DOES UNSUPERVISED DEEP LEARNING WORK? - A PERSPECTIVE FROM GROUP THEORY

Arnab Paul, Suresh Venkatasubramanian|arXiv (Cornell University)|Jan 1, 2015

Generative Adversarial Networks and Image Synthesis20 references3 citations

TL;DR

This paper introduces a group-theoretic framework to explain why unsupervised deep learning works, showing that pretraining in deep networks corresponds to searching for features with minimal group orbits—intuitively, the simplest features—thereby explaining why deep networks learn simple representations first. Repeating this process across layers captures increasingly complex, higher-order representations through the structure of shadow groups approximating neural network behavior.

ABSTRACT

Why does Deep Learning work? What representations does it capture? How do higher-order representations emerge? We study these questions from the perspective of group theory, thereby opening a new approach towards a theory of Deep learning. One factor behind the recent resurgence of the subject is a key algorithmic step called pretraining: first search for a good generative model for the input samples, and repeat the process one layer at a time. We show deeper implications of this simple principle, by establishing a connection with the interplay of orbits and stabilizers of group actions. Although the neural networks themselves may not form groups, we show the existence of shadow groups whose elements serve as close approximations. Over the shadow groups, the pretraining step, originally introduced as a mechanism to better initialize a network, becomes equivalent to a search for features with minimal orbits. Intuitively, these features are in a way the simplest. Which explains why a deep learning network learns simple features first. Next, we show how the same principle, when repeated in the deeper layers, can capture higher order representations, and why representation complexity increases as the layers get deeper.

Motivation & Objective

• To understand why unsupervised deep learning succeeds in learning meaningful representations.
• To explain the emergence of hierarchical, increasingly complex representations in deep networks.
• To provide a theoretical foundation for pretraining as a mechanism for feature discovery.
• To establish a connection between group actions (orbits and stabilizers) and the learning dynamics of deep neural networks.

Proposed method

• Introduce the concept of 'shadow groups'—approximate group structures that mirror the behavior of deep neural network layers.
• Model the pretraining process as a search for features with minimal group orbits, which correspond to the simplest, most invariant representations.
• Use the interplay between group orbits and stabilizers to formalize how features are selected and refined layer by layer.
• Show that repeated application of the pretraining step across layers leads to increasingly complex representations through the hierarchical structure of orbit minimization.
• Demonstrate that although neural networks do not form groups, their learning dynamics can be approximated by group-theoretic principles via shadow groups.

Experimental results

Research questions

• RQ1 Why do deep neural networks learn simple features before complex ones?
• RQ2 How does the pretraining process in unsupervised deep learning relate to group-theoretic structures?
• RQ3 What role do orbits and stabilizers of group actions play in the emergence of hierarchical representations?
• RQ4 How can the learning process in deeper layers be explained through the repeated minimization of orbit size?

Key findings

• Pretraining in deep networks corresponds to a search for features with minimal group orbits, which are the simplest and most invariant representations.
• The concept of 'shadow groups' provides a theoretical approximation of neural network behavior using group-theoretic principles, even when networks themselves are not groups.
• The repeated application of the pretraining step across layers leads to the emergence of higher-order representations through the hierarchical refinement of orbit structures.
• Features with minimal orbits are learned first because they are the most stable and invariant under transformation, explaining the observed inductive bias in deep learning.