[Paper Review] A Deep Neural Network's Loss Surface Contains Every Low-dimensional Pattern

Wojciech Marian Czarnecki, Simon Osindero+2 more|arXiv (Cornell University)|Dec 16, 2019

Neural Networks and Applications5 references4 citations

TL;DR

This paper proves that the loss surface of any sufficiently wide and deep neural network contains every possible low-dimensional smooth pattern—regardless of dataset or task—by constructing a universal approximation of arbitrary target functions within the loss landscape. The key result is that such patterns are not only embeddable but also optimizable via standard training, transferable to test sets, and locatable near global minima.

ABSTRACT

The work "Loss Landscape Sightseeing with Multi-Point Optimization" (Skorokhodov and Burtsev, 2019) demonstrated that one can empirically find arbitrary 2D binary patterns inside loss surfaces of popular neural networks. In this paper we prove that: (i) this is a general property of deep universal approximators; and (ii) this property holds for arbitrary smooth patterns, for other dimensionalities, for every dataset, and any neural network that is sufficiently deep and wide. Our analysis predicts not only the existence of all such low-dimensional patterns, but also two other properties that were observed empirically: (i) that it is easy to find these patterns; and (ii) that they transfer to other data-sets (e.g. a test-set).

Motivation & Objective

• To theoretically explain why arbitrary 2D binary patterns can be found in deep neural network loss surfaces, as empirically observed by Skorokhodov and Burtsev (2019).
• To establish that this property is not limited to binary or 2D patterns but extends to all smooth, bounded, low-dimensional patterns.
• To demonstrate that such patterns can be found with optimization complexity comparable to standard supervised training.
• To show that these patterns transfer from training to test sets and can be embedded near global minima.
• To investigate whether such patterns can be constructed independently of the network’s input-processing function, enabling geometric regularizations to be 'cheated' by the model.

Proposed method

• Construct a neural network with a fixed input projection and learnable bias vectors to encode the target pattern in the loss surface.
• Use the Universal Approximation Theorem to train a subnetwork to approximate a target function $\mathcal{T}: [0,1]^z \to [0,1]$ in a $z$-dimensional section of the parameter space.
• Decompose the loss into two parts: one for the main task (input-to-output mapping) and one for the pattern-encoding subnetwork, using a separate head per pattern dimension.
• Leverage the Lipschitz continuity of the loss function $\ell$ to bound the approximation error between the learned loss and the target pattern $\mathcal{T}$.
• Ensure the pattern-encoding subnetwork is trained independently from the input-processing path, preserving generalization and enabling transfer to test sets.
• Prove that the total loss $L(\mathbf{h})$ approximates $\mathcal{T}(\mathbf{h})$ within an error $\epsilon$ that vanishes as network width increases, using smoothness and surjectivity of activation functions.

Experimental results

Research questions

• RQ1 Can every smooth, bounded, low-dimensional pattern be embedded within the loss surface of a sufficiently wide and deep neural network?
• RQ2 Does the ability to embed such patterns generalize from training to test data, and if so, why?
• RQ3 Is finding these patterns via optimization as hard as standard supervised learning?
• RQ4 Can such patterns be embedded near the global minimum of the original training objective?
• RQ5 Can the network's loss surface geometry be manipulated to satisfy arbitrary local geometric constraints without affecting input-level performance?

Key findings

• The loss surface of any sufficiently wide and deep neural network contains every smooth, bounded, low-dimensional pattern $\mathcal{T}: [0,1]^z \to [0,1]$.
• The approximation error between the learned loss and the target pattern $\mathcal{T}$ can be made arbitrarily small by increasing network width, due to the Universal Approximation Theorem and Lipschitz continuity of the loss.
• The construction ensures that the embedded pattern is transferable from training to test sets, as the pattern is encoded in a separate, non-input-dependent subnetwork.
• Optimizing for such patterns is no more complex than standard supervised learning, as the problem reduces to training a standard neural network on a constructed target.
• The method allows for the embedding of patterns that are $\epsilon$-close to the global minimum of the original training objective, preserving the network's overall performance.
• The patterns can be made axis-aligned and are robust to changes in the input distribution, provided the loss function and label distribution $P(\mathbf{y})$ remain unchanged.