[Paper Review] The Gaussian equivalence of generative models for learning with shallow neural networks

Sebastian Goldt, Loureiro, Bruno+4 more|arXiv (Cornell University)|Jun 25, 2020

Generative Adversarial Networks and Image Synthesis97 references48 citations

TL;DR

This paper analyzes a teacher-student setup where data is generated by a latent Gaussian vector passed through a generative network, and studies learning with shallow neural networks under a Gaussian-equivalence framework.

ABSTRACT

Understanding the impact of data structure on the computational tractability of learning is a key challenge for the theory of neural networks. Many theoretical works do not explicitly model training data, or assume that inputs are drawn component-wise independently from some simple probability distribution. Here, we go beyond this simple paradigm by studying the performance of neural networks trained on data drawn from pre-trained generative models. This is possible due to a Gaussian equivalence stating that the key metrics of interest, such as the training and test errors, can be fully captured by an appropriately chosen Gaussian model. We provide three strands of rigorous, analytical and numerical evidence corroborating this equivalence. First, we establish rigorous conditions for the Gaussian equivalence to hold in the case of single-layer generative models, as well as deterministic rates for convergence in distribution. Second, we leverage this equivalence to derive a closed set of equations describing the generalisation performance of two widely studied machine learning problems: two-layer neural networks trained using one-pass stochastic gradient descent, and full-batch pre-learned features or kernel methods. Finally, we perform experiments demonstrating how our theory applies to deep, pre-trained generative models. These results open a viable path to the theoretical study of machine learning models with realistic data.

Motivation & Objective

• Motivate learning with generative data beyond i.i.d. assumptions in a teacher-student setting.
• Model data as x = G(c) with c ~ N(0,I_D) and labels from a two-layer teacher network.
• Derive closed-form learning rules for a two-layer neural network or a single layer after a fixed feature map.
• Investigate conditions under which Gaussian approximations capture the learning dynamics of generative data.

Proposed method

• Formulate the data-generating process as c ~ N(0,I_D) followed by x = G(c).
• Define labels y via a two-layer teacher network responding to c.
• Analyze learning with a two-layer neural network in closed form.
• Analyze learning with a single-layer network after projection through a fixed feature map.
• Possibly leverage Gaussian equivalence principles to relate generative models to standard Gaussian settings.

Experimental results

Research questions

• RQ1 Does Gaussian equivalence hold for learning with data generated by a deep generative network?
• RQ2 How does projection through a fixed feature map affect learnability of shallow networks?
• RQ3 What are the closed-form learning dynamics for two-layer and single-layer networks under this generative setup?
• RQ4 Under what conditions can generative data be treated as Gaussian for training shallow networks?

Key findings

• Establishes a Gaussian-equivalence perspective for generative-model-based data in shallow-network learning.
• Derives closed-form learning expressions for a two-layer network in the teacher-student setup.
• Shows how fixed feature-map projections influence learning with a single-layer network.
• Provides insights into when generative data can be analyzed using Gaussian-approximation tools.