[Paper Review] Control On the Manifolds Of Mappings As a Setting For Deep Learning.

Andrei Agrachev, Andrey Sarychev|arXiv (Cornell University)|Aug 28, 2020

Advanced Data Processing Techniques4 citations

TL;DR

This paper formulates deep learning training as an optimal control problem on manifolds of mappings, modeling deep neural networks as continuous-time control systems with infinitely many layers. It establishes approximate controllability of such systems on diffeomorphism groups of R^n, T^n, and S^2, demonstrating their potential for high-accuracy approximation of complex classification maps.

ABSTRACT

We use a control-theoretic setting to model the process of training (deep learning) of Artificial Neural Networks (ANN), which are aimed at solving classification problems. A successful classifier is the network whose input-output map approximates well the classifying map defined on a finite or an infinite training set. A fruitful idea is substitution of a multi-layer ANN by a continuous-time control system, which can be seen as a neural network with infinite number of layers. Under certain conditions it can achieve high rate of approximation with presumably not so high computational cost. The problem of best approximation for this model results in optimal control problem of Bolza type for ensembles of points. The two issues to be studied are: i) possibility of a satisfactory approximation of complex classification profiles; ii) finding the values of parameters (controls) which provide the best approximation. In control-theoretic terminology it corresponds respectively to the verification of an ensemble controllability property and to the solution of an ensemble optimal control problem. In the present contribution we concentrate on the first type of problems; our main results include examples of control systems, which are approximately controllable in the groups of diffeomorphisms of $\mathbb{R}^n, \mathbb{T}^n, \mathbb{S}^2$.

Motivation & Objective

• To model deep neural network training as a continuous-time control system with infinite layers.
• To investigate whether such control systems can approximate complex classification profiles with high accuracy.
• To verify ensemble controllability properties for mappings on R^n, T^n, and S^2.
• To lay the theoretical groundwork for efficient deep learning via control-theoretic approximation.

Proposed method

• Modeling multi-layer neural networks as continuous-time control systems on manifolds of diffeomorphisms.
• Formulating the training process as a Bolza-type optimal control problem for ensembles of points.
• Using control-theoretic tools to analyze controllability of the system on R^n, T^n, and S^2.
• Applying results from differential geometry and control theory to study approximation capabilities.
• Focusing on the ensemble controllability property as a necessary condition for effective approximation.
• Analyzing the structure of control systems that enable high-precision mapping of input-output relationships.

Experimental results

Research questions

• RQ1 Can continuous-time control systems on manifolds of diffeomorphisms achieve high-accuracy approximation of complex classification maps?
• RQ2 What conditions ensure that such control systems are approximately controllable on R^n, T^n, and S^2?
• RQ3 How does the control-theoretic formulation enable efficient approximation with potentially lower computational cost than discrete deep networks?
• RQ4 What types of control systems on these manifolds exhibit favorable approximation properties for classification tasks?
• RQ5 Is ensemble controllability a sufficient condition for effective learning in this framework?

Key findings

• The paper demonstrates that certain control systems are approximately controllable on the diffeomorphism group of R^n.
• It establishes approximate controllability for systems on the n-dimensional torus T^n.
• It proves approximate controllability for systems on the 2-sphere S^2.
• The results suggest that continuous-time control systems on these manifolds can effectively approximate complex classification profiles.
• The framework provides a theoretical basis for deep learning with potentially reduced computational cost through infinite-layer modeling.
• The analysis confirms that the ensemble controllability property is achievable for specific geometric settings, supporting the feasibility of the proposed approach.