[Paper Review] Optimization for deep learning: theory and algorithms

Ruoyu Sun|arXiv (Cornell University)|Dec 19, 2019

Stochastic Gradient Optimization Techniques256 references132 citations

TL;DR

This survey reviews optimization methods and theory for training neural networks, addressing gradient issues, training tricks, and both local and global training questions.

ABSTRACT

When and why can a neural network be successfully trained? This article provides an overview of optimization algorithms and theory for training neural networks. First, we discuss the issue of gradient explosion/vanishing and the more general issue of undesirable spectrum, and then discuss practical solutions including careful initialization and normalization methods. Second, we review generic optimization methods used in training neural networks, such as SGD, adaptive gradient methods and distributed methods, and theoretical results for these algorithms. Third, we review existing research on the global issues of neural network training, including results on bad local minima, mode connectivity, lottery ticket hypothesis and infinite-width analysis.

Motivation & Objective

• Explain why neural networks train successfully and what factors influence training success.
• Survey gradient explosion/vanishing issues and spectrum control with practical remedies.
• Review generic optimization algorithms used in neural networks and their theoretical results.
• Discuss global training challenges such as bad local minima, mode connectivity, lottery tickets, and infinite-width analyses.

Proposed method

• Discuss gradient explosion/vanishing and spectrum control; present remedies like careful initialization and normalization.
• Review backpropagation and provide a structured gradient computation framework.
• Summarize generic optimization methods for non-convex problems including SGD, adaptive methods, and distributed training, with convergence insights.
• Present discussions on neural-network specific tricks and their theoretical bases.
• Examine global optimization perspectives including landscape properties and infinite-width analysis.

Experimental results

Research questions

• RQ1 What optimization challenges arise in training deep neural networks and how can they be mitigated?
• RQ2 How do initialization, normalization, and architectural choices influence convergence and training speed?
• RQ3 What are the theoretical guarantees and limitations of gradient-based methods for deep learning?
• RQ4 What global properties of neural networks affect the ability to find good solutions (e.g., local minima, mode connectivity, lottery tickets, NTK)?

Key findings

• Gradient issues such as explosion and vanishing are central to training difficulty and are linked to convergence speed and landscape properties.
• Careful initialization and normalization play crucial roles in stabilizing training and enabling convergence.
• SGD and adaptive methods, along with distributed training, are central optimization tools with established convergence and complexity results under certain assumptions.
• Global optimization perspectives reveal phenomena like mode connectivity and infinite-width behaviors that inform understanding of training dynamics.
• Theoretical analyses connect initialization, signal propagation, and width to practical training success across various activation functions.