[Paper Review] A Recipe for Global Convergence Guarantee in Deep Neural Networks

Kenji Kawaguchi, Qingyun Sun|arXiv (Cornell University)|Apr 12, 2021

Stochastic Gradient Optimization Techniques46 references4 citations

TL;DR

This paper proposes a two-phase training algorithm that guarantees global convergence for practical deep neural networks beyond the neural tangent kernel (NTK) regime by introducing a verifiable data- and architecture-dependent 'expressivity condition'. The method ensures all layers converge to global minima under this condition, with theoretical verification for narrow-deep fully connected networks and numerical validation for deep ResNets with batch normalization on standard datasets, while preserving generalization performance comparable to standard SGD.

ABSTRACT

Existing global convergence guarantees of (stochastic) gradient descent do not apply to practical deep networks in the practical regime of deep learning beyond the neural tangent kernel (NTK) regime. This paper proposes an algorithm, which is ensured to have global convergence guarantees in the practical regime beyond the NTK regime, under a verifiable condition called the expressivity condition. The expressivity condition is defined to be both data-dependent and architecture-dependent, which is the key property that makes our results applicable for practical settings beyond the NTK regime. On the one hand, the expressivity condition is theoretically proven to hold data-independently for fully-connected deep neural networks with narrow hidden layers and a single wide layer. On the other hand, the expressivity condition is numerically shown to hold data-dependently for deep (convolutional) ResNet with batch normalization with various standard image datasets. We also show that the proposed algorithm has generalization performances comparable with those of the heuristic algorithm, with the same hyper-parameters and total number of iterations. Therefore, the proposed algorithm can be viewed as a step towards providing theoretical guarantees for deep learning in the practical regime.

Motivation & Objective

• To bridge the gap between theoretical global convergence guarantees and practical deep learning, where existing methods fail beyond the NTK regime.
• To develop a training algorithm that enables representation learning in deep networks while guaranteeing global convergence for all layers.
• To define and verify a condition—expressivity condition—that is both data-dependent and architecture-dependent, enabling applicability to real-world networks.
• To demonstrate that the proposed method maintains generalization performance comparable to standard SGD with identical hyperparameters and training iterations.

Proposed method

• The proposed algorithm modifies any first-order training method into a two-phase training procedure to ensure global convergence.
• The key innovation is the introduction of the expressivity condition, which is both data-dependent and architecture-dependent, ensuring global convergence when satisfied.
• The method relies on theoretical analysis showing that if the expressivity condition holds, then gradient descent converges globally to the global minimum for all network layers.
• For theoretical verification, the expressivity condition is proven to hold for fully connected networks with narrow hidden layers and one wide final layer.
• For empirical validation, the condition is numerically verified to hold in deep ResNets with batch normalization on standard image datasets such as CIFAR-10 and ImageNet.
• The algorithm preserves generalization performance by matching the test accuracy of standard SGD when using the same hyperparameters and total training iterations.

Experimental results

Research questions

• RQ1 Can global convergence be guaranteed for practical deep neural networks outside the NTK regime?
• RQ2 What condition enables global convergence in the practical regime, and is it verifiable for real-world architectures?
• RQ3 Does the proposed method maintain generalization performance comparable to standard training procedures?
• RQ4 Can the expressivity condition be theoretically proven to hold for fully connected networks with specific width configurations?
• RQ5 Can the expressivity condition be numerically verified for deep convolutional networks with batch normalization on standard datasets?

Key findings

• The expressivity condition is theoretically proven to hold for fully connected deep neural networks with narrow hidden layers and a single wide final layer, ensuring global convergence.
• The expressivity condition is numerically verified to hold for deep ResNets with batch normalization on CIFAR-10, CIFAR-100, and ImageNet datasets.
• The two-phase training algorithm achieves global convergence for all layers when the expressivity condition is satisfied, under standard assumptions on loss and gradient Lipschitz continuity.
• The generalization performance of the two-phase algorithm is empirically shown to be comparable to standard SGD with the same hyperparameters and total number of iterations.
• The method successfully enables representation learning in deep networks, unlike previous NTK-based approaches that remain in the lazy training regime.
• The theoretical analysis shows that if the rank of the Jacobian matrix equals the number of output units, and the expressivity condition holds, then global convergence is guaranteed.