[Paper Review] On the Turnpike to Design of Deep Neural Nets: Explicit Depth Bounds

Timm Faulwasser, Arne-Jens Hempel+1 more|arXiv (Cornell University)|Jan 8, 2021

Model Reduction and Neural Networks25 references4 citations

TL;DR

This paper proposes a novel optimal control framework to derive explicit depth bounds for deep neural networks (DNNs) by leveraging turnpike and dissipativity properties. By modeling DNN training as a finite-horizon optimal control problem and enforcing strict dissipativity via regularization, the authors derive analytically computable depth bounds that are numerically validated on the Two Spiral Task, showing non-conservative estimates, especially when using ℓ∞-norm regularization.

ABSTRACT

It is well-known that the training of Deep Neural Networks (DNN) can be formalized in the language of optimal control. In this context, this paper leverages classical turnpike properties of optimal control problems to attempt a quantifiable answer to the question of how many layers should be considered in a DNN. The underlying assumption is that the number of neurons per layer -- i.e., the width of the DNN -- is kept constant. Pursuing a different route than the classical analysis of approximation properties of sigmoidal functions, we prove explicit bounds on the required depths of DNNs based on asymptotic reachability assumptions and a dissipativity-inducing choice of the regularization terms in the training problem. Numerical results obtained for the two spiral task data set for classification indicate that the proposed estimates can provide non-conservative depth bounds.

Motivation & Objective

• To address the long-standing empirical challenge of determining optimal DNN depth by framing it as a quantifiable design problem.
• To provide explicit, analytically derived depth bounds for DNNs under constant width and asymptotic reachability assumptions.
• To establish a constructive method for designing regularization terms that induce strict dissipativity in the training OCP.
• To evaluate the tightness and practical relevance of the derived bounds using numerical experiments on classification tasks.
• To bridge systems theory and deep learning by applying optimal control tools—particularly turnpike and dissipativity theory—to DNN architecture design.

Proposed method

• Formulates the training of a feed-forward DNN as a finite-horizon optimal control problem (OCP), treating depth as discrete time steps.
• Applies turnpike theory to identify a central, stable regime in the OCP solution, which corresponds to the network's optimal training path.
• Imposes a strict dissipativity condition on the OCP by designing a regularization term in the stage cost, ensuring the existence of a turnpike.
• Derives explicit depth bounds using asymptotic reachability assumptions and the dissipativity-induced structure of the OCP.
• Employs two types of state penalization—ℓ2 and ℓ∞—to compute a-posteriori depth estimates, with ℓ∞ yielding significantly tighter bounds.
• Validates results numerically on the Two Spiral Task, comparing bounds with actual training performance and convergence.

Experimental results

Research questions

• RQ1 Can turnpike and dissipativity theory be used to derive explicit, non-conservative depth bounds for DNNs?
• RQ2 How does the choice of regularization in the OCP stage cost affect the derived depth bounds?
• RQ3 Are the derived depth bounds tight and practically meaningful, especially when compared to empirical training behavior?
• RQ4 Can ℓ∞-norm regularization yield significantly tighter depth estimates than ℓ2, particularly as dataset size increases?
• RQ5 To what extent is asymptotic reachability of the target state sufficient for successful classification in DNNs?

Key findings

• The proposed method yields explicit depth bounds for DNNs based on dissipativity and turnpike theory, marking the first such analytical derivation to our knowledge.
• Using ℓ∞-norm regularization in the stage cost leads to significantly tighter depth estimates, ranging from 6.31 to 11.06 for the Two Spiral Task, even as dataset size increases.
• In contrast, ℓ2-based bounds grow substantially with dataset size, reaching up to 6,830 for D=500, indicating poor scalability under standard regularization.
• Numerical results show that the a-posteriori depth estimates are not overly conservative, with the best estimates aligning closely with observed training behavior.
• The ℓ∞-norm-based bound remains stable and centered around 7–8 on average, even with noisy data, suggesting robustness to data perturbations.
• The study reveals that reachability of the target state is sufficient but not necessary for classification, indicating room for more general formulations in future work.