[Paper Review] On-line learning dynamics of ReLU neural networks using statistical physics techniques

Michiel Straat, Michael Biehl|arXiv (Cornell University)|Mar 18, 2019

Neural Networks and Applications11 references4 citations

TL;DR

This paper derives exact macroscopic on-line learning dynamics for two-layer ReLU neural networks using statistical physics techniques, formulating a system of ordinary differential equations (ODEs) for order parameters in the thermodynamic limit. The key contribution is revealing distinct learning behaviors in ReLU networks—such as symmetric plateaus and weight sharing in overrealizable settings—compared to sigmoidal networks, with critical learning rate $η_c = 2$ governing stability and specialization.

ABSTRACT

We introduce exact macroscopic on-line learning dynamics of two-layer neural networks with ReLU units in the form of a system of differential equations, using techniques borrowed from statistical physics. For the first experiments, numerical solutions reveal similar behavior compared to sigmoidal activation researched in earlier work. In these experiments the theoretical results show good correspondence with simulations. In ove-rrealizable and unrealizable learning scenarios, the learning behavior of ReLU networks shows distinctive characteristics compared to sigmoidal networks.

Motivation & Objective

• To develop an exact macroscopic description of on-line learning dynamics in two-layer ReLU neural networks using statistical physics methods.
• To understand how ReLU activation influences learning dynamics compared to sigmoidal activation, particularly in overrealizable and unrealizable scenarios.
• To identify the role of symmetry, fixed points, and repulsive dynamics in the emergence of specialization in ReLU networks.
• To derive a critical learning rate $\eta_c = 2$ that determines stability and convergence behavior in the thermodynamic limit.
• To compare ReLU and sigmoidal (e.g., Erf) activation behaviors in terms of weight sharing, generalization error, and convergence to optimal solutions.

Proposed method

• Uses the thermodynamic limit $N \to \infty$ and central limit theorem (CLT) to derive macroscopic order parameters (e.g., $R_{in}, Q_{ik}, T_{nm}$) from microscopic weights.
• Applies on-line gradient descent with learning rate $\eta$ scaled by input dimension $N$, leading to stochastic update rules for weight vectors.
• Derives a closed-form system of ODEs for the evolution of order parameters using joint Gaussian statistics of pre-activations and ReLU nonlinearities.
• Employs the inverse of the multivariate Gaussian cumulative distribution function to compute expectations of the form $\langle \theta(u) v w \theta(w) \rangle$.
• Performs linear stability analysis on the ODE system to identify fixed points and determine the critical learning rate $\eta_c = 2$.
• Validates theoretical predictions via numerical simulations with $N = 10^4$, comparing generalization error and order parameter evolution.

Experimental results

Research questions

• RQ1 How do the macroscopic learning dynamics of ReLU networks differ from those of sigmoidal networks in on-line learning?
• RQ2 What causes the emergence of symmetric plateaus in ReLU network training, and what drives the transition to specialization?
• RQ3 How does the critical learning rate $\eta_c = 2$ affect the stability and convergence of ReLU network learning?
• RQ4 In overrealizable settings, why do multiple ReLU units learn the same teacher unit through weight sharing, and what is the role of the piecewise linear nature of ReLU?
• RQ5 How does the generalization error evolve in ReLU networks, and what determines whether it converges to zero in overrealizable vs. unrealizable cases?

Key findings

• The theoretical ODE system for ReLU networks shows excellent agreement with simulations for $N = 10^4$, validating the macroscopic approximation.
• A suboptimal plateau in generalization error emerges due to a repulsive fixed point, with $R_{in} \approx 0.52$ for $K=M=2$, indicating symmetric weight distribution before specialization.
• Specialization is driven by a positive eigenvalue $\lambda_5 = 0.24$ in the linearized system, causing asymmetric weight updates toward distinct teacher units.
• In overrealizable settings ($K > M$), ReLU units can collectively learn a single teacher unit via weight sharing: $\bm{J}_2 + \bm{J}_3 = \bm{B}_2$ for $a = b = 0.5$, with infinitely many such solutions.
• For $K=3, M=2$, ReLU networks achieve perfect generalization error $\epsilon_g(\infty) = 0$ with $Q_{11}(\infty) = 1.00$, $Q_{22}(\infty) = Q_{33}(\infty) \approx 0.25$, and $R_{22} \approx R_{32} \approx 0.5$.
• In contrast, sigmoidal (Erf) networks eliminate redundant units via $Q_{22}(\infty) = 0$, showing no weight sharing, and converge to $\epsilon_g(\infty) = 0$ only when $K=M$.