[Paper Review] Representing smooth functions as compositions of near-identity functions with implications for deep network optimization

Peter L. Bartlett, Steven N. Evans+1 more|arXiv (Cornell University)|Apr 13, 2018

Stochastic Gradient Optimization Techniques13 references16 citations

TL;DR

This paper proves that any smooth bi-Lipschitz function can be exactly represented as a composition of near-identity functions, each with a small Lipschitz deviation from the identity. It further shows that in nonlinear regression, functional gradient descent over such compositions avoids suboptimal critical points, unlike parametric gradient methods in standard residual networks with sigmoid activations.

ABSTRACT

We show that any smooth bi-Lipschitz $h$ can be represented exactly as a composition $h_m \circ ... \circ h_1$ of functions $h_1,...,h_m$ that are close to the identity in the sense that each $\left(h_i-\mathrm{Id} ight)$ is Lipschitz, and the Lipschitz constant decreases inversely with the number $m$ of functions composed. This implies that $h$ can be represented to any accuracy by a deep residual network whose nonlinear layers compute functions with a small Lipschitz constant. Next, we consider nonlinear regression with a composition of near-identity nonlinear maps. We show that, regarding Fréchet derivatives with respect to the $h_1,...,h_m$, any critical point of a quadratic criterion in this near-identity region must be a global minimizer. In contrast, if we consider derivatives with respect to parameters of a fixed-size residual network with sigmoid activation functions, we show that there are near-identity critical points that are suboptimal, even in the realizable case. Informally, this means that functional gradient methods for residual networks cannot get stuck at suboptimal critical points corresponding to near-identity layers, whereas parametric gradient methods for sigmoidal residual networks suffer from suboptimal critical points in the near-identity region.

Motivation & Objective

• To establish a theoretical foundation for deep residual networks by showing smooth bi-Lipschitz functions can be exactly represented as compositions of near-identity functions.
• To analyze the optimization landscape of nonlinear regression using compositions of near-identity maps, focusing on critical points.
• To compare functional gradient methods (with respect to the functions) with parametric gradient methods (with respect to network weights) in residual networks.
• To explain why functional gradient descent avoids suboptimal critical points in the near-identity region, while parametric methods do not.

Proposed method

• Proves that any smooth bi-Lipschitz map $ h $ can be exactly decomposed as $ h = h_m \circ \cdots \circ h_1 $, where each $ h_i = \mathrm{Id} + N_i $ and $ \|N_i\|_L \leq C/m $ for some constant $ C $, with $ m $ the number of layers.
• Uses the gradient theorem for line integrals and smoothness assumptions to bound the deviation of $ h $ from its linear approximation, establishing the Lipschitz control on $ N_i $.
• Analyzes the Fréchet derivative of the quadratic loss with respect to the functions $ h_i $, showing that any critical point in the near-identity region must be a global minimizer.
• Constructs a counterexample using a two-layer residual network with sigmoid activations to show that parametric gradient descent can converge to suboptimal critical points even in the realizable case.
• Applies results from functional analysis and non-convex optimization to compare the behavior of infinite-parameter (nonparametric) and finite-parameter (parametric) optimization in residual networks.
• Uses regularization or early stopping to maintain near-identity behavior and links this to the avoidance of bad critical points in functional gradient methods.

Experimental results

Research questions

• RQ1 Can any smooth bi-Lipschitz function be exactly represented as a composition of near-identity functions with diminishing Lipschitz deviation per layer?
• RQ2 Does functional gradient descent over the space of near-identity maps avoid suboptimal critical points in nonlinear regression?
• RQ3 Why do parametric gradient methods in standard residual networks with sigmoid activations fail to avoid suboptimal critical points in the near-identity region?
• RQ4 What is the relationship between the expressiveness of deep residual networks and the optimization landscape under functional versus parametric parameterization?
• RQ5 How does maintaining near-identity behavior through regularization or early stopping affect the convergence properties of residual network optimization?

Key findings

• Any smooth bi-Lipschitz function $ h $ can be exactly represented as a composition of $ m $ functions $ h_i = \mathrm{Id} + N_i $, where $ \|N_i\|_L \leq C/m $, ensuring each $ N_i $ is a small perturbation.
• In nonlinear regression, any critical point of the quadratic loss with respect to the functions $ h_i $ (under Fréchet derivatives) must be a global minimizer, implying no suboptimal critical points exist in the near-identity region.
• In contrast, for a parametric residual network with sigmoid activations, there exist regression problems where gradient descent converges to suboptimal critical points even when the data is realizable.
• The functional gradient method avoids bad critical points due to inter-layer interactions, while parametric methods are vulnerable to such issues due to internal dynamics within each layer.
• The result implies that functional gradient descent in deep residual networks with near-identity layers is globally convergent under the given parameterization, unlike standard parametric training.
• Regularization that enforces small Lipschitz deviation from identity can prevent overfitting and improve optimization by enabling a large number of effective directions in function space.