[论文解读] A mean-field limit for certain deep neural networks

Dyego Carlos Souza Anacleto de Araújo, Roberto I. Oliveira+1 more|arXiv (Cornell University)|Jun 1, 2019

Stochastic Gradient Optimization Techniques参考文献 31被引用 39

一句话总结

本文推导了一个均值场（McKean-Vlasov）极限，描述深度神经网络在 L≥3、宽度很大 N、输入和输出处的固定随机特征附近的训练动力学。它表明权重表现得像理想粒子，其分布由均值场模型控制，并且在该设定下证明了 McKean-Vlasov 问题的存在性和唯一性。

ABSTRACT

Understanding deep neural networks (DNNs) is a key challenge in the theory of machine learning, with potential applications to the many fields where DNNs have been successfully used. This article presents a scaling limit for a DNN being trained by stochastic gradient descent. Our networks have a fixed (but arbitrary) number $L\geq 2$ of inner layers; $N\gg 1$ neurons per layer; full connections between layers; and fixed weights (or "random features" that are not trained) near the input and output. Our results describe the evolution of the DNN during training in the limit when $N o +\infty$, which we relate to a mean field model of McKean-Vlasov type. Specifically, we show that network weights are approximated by certain "ideal particles" whose distribution and dependencies are described by the mean-field model. A key part of the proof is to show existence and uniqueness for our McKean-Vlasov problem, which does not seem to be amenable to existing theory. Our paper extends previous work on the $L=1$ case by Mei, Montanari and Nguyen; Rotskoff and Vanden-Eijnden; and Sirignano and Spiliopoulos. We also complement recent independent work on $L>1$ by Sirignano and Spiliopoulos (who consider a less natural scaling limit) and Nguyen (who nonrigorously derives similar results).

研究动机与目标

• Motivate a mean-field scaling approach to understand how deep neural networks evolve when trained by SGD.
• Extend shallow-network mean-field results to deep architectures with layered path-wise dependencies.
• Describe a rigorous McKean-Vlasov framework that captures layer-dependent weight distributions and their interactions along input–output paths.
• Establish existence and uniqueness for the resulting McKean-Vlasov problem and relate SGD dynamics to a continuous-time gradient flow.

提出的方法

• Introduce a deep network model with L≥3 hidden layers, N neurons per layer, full connections, and frozen random features at input and output.
• Formulate an ansatz that weights along input–output paths behave as interacting particles whose law is described by a mean-field measure.
• Derive mean-field representations of neuron values and gradients along network paths, leading to a McKean-Vlasov evolution.
• Prove existence and uniqueness for the McKean-Vlasov problem arising in the deep-network mean-field limit.
• Connect SGD updates to a continuous-time gradient flow in the mean-field setting.
• Compare with related works and discuss the scaling and time-scale implications for different layers.

实验结果

研究问题

• RQ1 What is the appropriate mean-field scaling for deep neural networks with many neurons per layer and fixed input/output features?
• RQ2 How do layer-dependent weight distributions evolve under SGD in the large-N limit, and do they exhibit propagation of chaos or path-wise dependencies?
• RQ3 Can one formulate and solve a McKean-Vlasov problem that accurately describes the learning dynamics of deep nets?
• RQ4 What are the relationships between SGD dynamics, ideal particle representations, and continuous-time gradient flows in this mean-field regime?
• RQ5 How does this deep-network mean-field limit extend existing shallow-network results and relate to other scaling limits in the literature?

主要发现

• Weights in deep networks with large width converge to distributions described by a McKean-Vlasov process, capturing layer-dependent and path-structured dependencies.
• The analysis introduces an ansatz based on input–output paths where ideal particles and their measures govern the dynamics.
• Existence and uniqueness of the McKean-Vlasov problem are established under the proposed framework.
• The gradients and loss can be approximated by mean-field quantities tied to the path measures, connecting SGD to a continuous-time gradient flow.
• The work extends prior shallow-network results to deeper architectures and clarifies the role of random features near input and output.
• The results complement related independent work and discuss differences in scaling and time-scale considerations.