[Paper Review] Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective

Guan-Horng Liu, Evangelos A. Theodorou|arXiv (Cornell University)|Aug 28, 2019

Gaussian Processes and Bayesian Inference107 references47 citations

TL;DR

A theoretical survey framing deep learning through dynamical systems and mean-field optimal control, linking DNN propagation and training dynamics to stability, optimization, and generalization.

ABSTRACT

Attempts from different disciplines to provide a fundamental understanding of deep learning have advanced rapidly in recent years, yet a unified framework remains relatively limited. In this article, we provide one possible way to align existing branches of deep learning theory through the lens of dynamical system and optimal control. By viewing deep neural networks as discrete-time nonlinear dynamical systems, we can analyze how information propagates through layers using mean field theory. When optimization algorithms are further recast as controllers, the ultimate goal of training processes can be formulated as an optimal control problem. In addition, we can reveal convergence and generalization properties by studying the stochastic dynamics of optimization algorithms. This viewpoint features a wide range of theoretical study from information bottleneck to statistical physics. It also provides a principled way for hyper-parameter tuning when optimal control theory is introduced. Our framework fits nicely with supervised learning and can be extended to other learning problems, such as Bayesian learning, adversarial training, and specific forms of meta learning, without efforts. The review aims to shed lights on the importance of dynamics and optimal control when developing deep learning theory.

Motivation & Objective

• Provide a unified dynamical-systems and optimal-control framework for understanding deep neural networks.
• Analyze information propagation through DNN layers using mean-field theory.
• Recast optimization algorithms as controllers to formulate training as an optimal control problem.
• Investigate convergence and generalization through stochastic dynamics of optimization methods.
• Discuss extensions to other learning paradigms and implications for architecture and hyper-parameter design.

Proposed method

• Model DNNs as discrete-time nonlinear dynamical systems with layers as time steps.
• Use mean-field theory to describe distributions of activations and pre-activations across layers and derive deterministic layer-wise dynamics.
• Define and analyze stability via Jacobian-like eigenvalues (chi_q* and chi_c*) and depth-scale xi_q* and xi_c* to distinguish ordered, critical, and chaotic phases.
• Relate training dynamics to mean-field Gram/NTK representations to show conditions for global convergence of gradient-based methods.
• Formulate DNN training as an optimal control problem, including mean-field extensions, to derive Pontryagin’s Minimum Principle (PMP) and connections to backpropagation.
• Discuss links to dynamic programming/HJB and show how optimization hyper-parameters can be optimized using control theory.

Experimental results

Research questions

• RQ1 How can DNN training and information propagation be analyzed within a dynamical systems framework?
• RQ2 What initialization and depth conditions ensure trainability and information flow through deep networks?
• RQ3 Can optimization algorithms for training be precisely recast as controllers, enabling principled design via optimal control theory?
• RQ4 How do mean-field approximations connect to convergence and generalization properties of over-parameterized networks?
• RQ5 What are the extensions of this framework to other learning paradigms like Bayesian learning, adversarial training, or meta-learning?

Key findings

• Mean-field analysis yields deterministic layer-wise dynamics for activations and pre-activations, enabling global characterization of information propagation.
• A phase diagram with a critical line separates ordered (stable, trainable) and chaotic (unstable) regimes, predicting trainability as a function of weight and bias variance.
• Under wide conditions, over-parameterized networks maintain initialization-like statistics during training, supporting global convergence results linked to Gram/NTK dynamics.
• The Gram matrix/NTK perspective shows linear convergence to the global minimum under sufficiently large width when the minimal eigenvalue of the NTK is bounded away from zero.
• Training can be formulated as a mean-field optimal control problem, enabling PMP and HJB-based analyses and suggesting adaptive hyper-parameter strategies from control theory.
• Backpropagation emerges as a discrete-time manifestation of PMP, linking dynamical systems theory directly to standard neural network training procedures.