[Paper Review] Continuous vs. Discrete Optimization of Deep Neural Networks

Omer Elkabetz, Nadav Cohen|arXiv (Cornell University)|Jul 14, 2021

Stochastic Gradient Optimization Techniques61 references4 citations

TL;DR

This paper establishes a formal link between continuous gradient flow and discrete gradient descent in deep neural networks, showing that under homogeneous activations, gradient flow trajectories are well-approximated by gradient descent due to favorable curvature. The key contribution is a convergence guarantee for gradient descent to the global minimum in deep linear networks under random initialization, derived via translation from continuous analysis.

ABSTRACT

Existing analyses of optimization in deep learning are either continuous, focusing on (variants of) gradient flow, or discrete, directly treating (variants of) gradient descent. Gradient flow is amenable to theoretical analysis, but is stylized and disregards computational efficiency. The extent to which it represents gradient descent is an open question in the theory of deep learning. The current paper studies this question. Viewing gradient descent as an approximate numerical solution to the initial value problem of gradient flow, we find that the degree of approximation depends on the curvature around the gradient flow trajectory. We then show that over deep neural networks with homogeneous activations, gradient flow trajectories enjoy favorable curvature, suggesting they are well approximated by gradient descent. This finding allows us to translate an analysis of gradient flow over deep linear neural networks into a guarantee that gradient descent efficiently converges to global minimum almost surely under random initialization. Experiments suggest that over simple deep neural networks, gradient descent with conventional step size is indeed close to gradient flow. We hypothesize that the theory of gradient flows will unravel mysteries behind deep learning.

Motivation & Objective

• To formally analyze the discrepancy between continuous gradient flow and discrete gradient descent in deep learning.
• To determine under what conditions gradient descent closely approximates gradient flow.
• To translate convergence guarantees from continuous gradient flow analysis into discrete gradient descent settings.
• To provide the first efficient convergence guarantee for gradient descent in deep (3+ layer) fixed-size networks with random initialization.
• To empirically validate that gradient descent trajectories remain close to gradient flow under standard training settings.

Proposed method

• Treats gradient descent as a numerical approximation to the initial value problem of gradient flow.
• Applies a fundamental theorem from numerical analysis to bound the approximation error by curvature, particularly the minimal Hessian eigenvalue along the trajectory.
• Analyzes deep linear networks with homogeneous activations (e.g., ReLU) under near-zero initialization.
• Uses the favorable curvature of gradient flow trajectories—characterized by a minimal Hessian eigenvalue bounded away from zero—to ensure tight approximation by gradient descent.
• Translates existing convergence results for gradient flow in deep linear networks into discrete convergence guarantees for gradient descent.
• Employs Frobenius norm inequalities and perturbation analysis to bound the deviation between discrete and continuous trajectories.

Experimental results

Research questions

• RQ1 How well does gradient descent approximate gradient flow in deep neural networks?
• RQ2 What role does curvature—specifically the minimal Hessian eigenvalue—play in determining the quality of the approximation between gradient descent and gradient flow?
• RQ3 Can convergence results derived for continuous gradient flow be rigorously transferred to discrete gradient descent in deep networks?
• RQ4 Under what conditions does gradient descent with finite step size efficiently converge to the global minimum in deep linear networks?
• RQ5 To what extent do empirical trajectories of gradient descent remain close to gradient flow in simple deep learning setups?

Key findings

• Gradient descent with moderately small step size closely approximates gradient flow in deep neural networks with homogeneous activations due to favorable curvature along the trajectory.
• The minimal Hessian eigenvalue along gradient flow trajectories in deep linear networks remains bounded away from zero, especially near convergence, ensuring strong approximation.
• The paper provides the first convergence guarantee for gradient descent in deep (3+ layer) fixed-size networks to reach the global minimum efficiently under random, data-independent initialization.
• Empirical results show that reducing step size leads to only slight changes in gradient descent trajectories, supporting the theoretical approximation claim.
• The approximation error between gradient descent and gradient flow is exponentially sensitive to the minimal Hessian eigenvalue, with other factors (smoothness, Lipschitz) having weaker influence.
• The derived bound on the deviation between discrete and continuous weights is proportional to √(n⁵ϵ)/(1−√(n⁵ϵ)), where ϵ controls initialization noise and n is the network depth.