[Paper Review] Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets

Rohith Kuditipudi, Xiang Wang+6 more|arXiv (Cornell University)|Jan 1, 2019

Stochastic Gradient Optimization Techniques29 citations

TL;DR

This paper explains the mode connectivity phenomenon in deep neural networks—where optimal solutions are connected by low-loss paths—by leveraging generic properties like dropout and noise stability in well-trained multilayer networks. The authors provide a theoretical framework showing that such connectivity arises naturally under realistic conditions, verified through experiments on practical deep nets.

ABSTRACT

Mode connectivity is a surprising phenomenon in the loss landscape of deep nets. Optima---at least those discovered by gradient-based optimization---turn out to be connected by simple paths on which the loss function is almost constant. Often, these paths can be chosen to be piece-wise linear, with as few as two segments. We give mathematical explanations for this phenomenon, assuming generic properties (such as dropout stability and noise stability) of well-trained deep nets, which have previously been identified as part of understanding the generalization properties of deep nets. Our explanation holds for realistic multilayer nets, and experiments are presented to verify the theory.

Motivation & Objective

• To understand why optimal solutions in deep neural networks are connected by low-loss paths, a phenomenon known as mode connectivity.
• To explain this connectivity through generic, empirically observed properties of well-trained deep nets, such as dropout stability and noise stability.
• To provide a theoretical foundation for mode connectivity that applies to realistic multilayer architectures, not just simplified models.
• To validate the theory with experiments on standard deep learning setups, confirming the existence of simple, low-loss paths between optima.

Proposed method

• Theoretical analysis is conducted under assumptions of dropout stability and noise stability, common in well-trained deep networks.
• The authors derive conditions under which two optimal models can be connected via piecewise linear paths with nearly constant loss.
• The analysis focuses on the geometry of the loss landscape, particularly the behavior of loss along linear interpolations between trained models.
• The framework applies to multilayer feedforward networks with standard activation functions and weight initialization.
• Theoretical results are supported by empirical validation on standard deep learning benchmarks, demonstrating low-loss paths with minimal segments.

Experimental results

Research questions

• RQ1 Why do optimal solutions in deep neural networks tend to be connected by low-loss paths in the loss landscape?
• RQ2 What underlying properties of well-trained deep nets give rise to this mode connectivity?
• RQ3 Can mode connectivity be explained through generic, empirically observed properties like dropout and noise stability?
• RQ4 Does the theoretical explanation hold for realistic multilayer network architectures?

Key findings

• Mode connectivity in deep nets arises naturally from generic properties such as dropout stability and noise stability, rather than being an artifact of specific architectures.
• Optimal solutions can be connected by piecewise linear paths with as few as two segments, maintaining nearly constant loss throughout.
• The theoretical framework applies to realistic multilayer networks, not just simplified or idealized settings.
• Experiments confirm that such low-loss paths exist between optima found via standard training procedures.