[Paper Review] The loss surface and expressivity of deep convolutional neural networks

Quynh Nguyen, Matthias Hein|arXiv (Cornell University)|Oct 30, 2017

Adversarial Robustness in Machine Learning65 references18 citations

TL;DR

This paper analyzes the loss surface and expressivity of wide, practical deep convolutional neural networks (CNNs) with shared weights and max pooling. It proves that when a CNN layer has more neurons than training samples, it produces linearly independent features, and for such wide networks, almost all critical points of the empirical loss are global minima with zero training error—indicating a well-behaved optimization landscape due to width.

ABSTRACT

We analyze the expressiveness and loss surface of practical deep convolutional neural networks (CNNs) with shared weights and max pooling layers. We show that such CNNs produce linearly independent features at a layer which has more neurons than the number of training samples. This condition holds e.g. for the VGG network. Furthermore, we provide for such wide CNNs necessary and sufficient conditions for global minima with zero training error. For the case where the wide layer is followed by a fully connected layer, we show that almost every critical point of the empirical loss is a global minimum with zero training error. Our analysis suggests that both depth and width are very important in deep learning. While depth brings more representational power and allows the network to learn high level features, width smoothes the optimization landscape of the loss function in the sense that a sufficiently wide network has a well-behaved loss surface with potentially no bad local minima.

Motivation & Objective

• To understand the expressiveness and optimization properties of practical deep CNNs with shared weights and max pooling.
• To investigate how network width affects the loss surface and the existence of global minima.
• To identify conditions under which wide CNNs achieve zero training error with global minima.
• To clarify the complementary roles of depth and width in deep learning.

Proposed method

• Theoretical analysis of CNNs with shared weights and max pooling layers under the condition that a hidden layer has more neurons than training samples.
• Derivation of necessary and sufficient conditions for global minima with zero training error in wide CNNs.
• Proof that for wide layers followed by fully connected layers, almost every critical point is a global minimum with zero training error.
• Use of linear algebra and optimization theory to analyze the structure of the loss surface in wide networks.
• Focus on the interplay between network width and the geometry of the empirical loss function.

Experimental results

Research questions

• RQ1 Under what conditions do wide CNNs achieve zero training error with global minima?
• RQ2 How does network width influence the geometry of the loss surface in CNNs?
• RQ3 What is the role of width versus depth in determining the optimization landscape of deep networks?
• RQ4 Why do wide CNNs exhibit fewer bad local minima compared to narrower networks?

Key findings

• When a CNN layer has more neurons than the number of training samples, it produces linearly independent features.
• For wide CNNs with more neurons than training samples, necessary and sufficient conditions for global minima with zero training error are derived.
• In wide CNNs where a wide layer is followed by a fully connected layer, almost every critical point of the empirical loss is a global minimum with zero training error.
• Network width significantly smooths the loss surface, reducing the presence of bad local minima.