[Paper Review] The Difficulty of Training Sparse Neural Networks

Utku Evci, Fabian Pedregosa+2 more|arXiv (Cornell University)|Jun 25, 2019

Advanced Neural Network Applications25 references32 citations

TL;DR

The paper analyzes why training sparse neural networks from scratch or via pruning yields different outcomes, showing monotonic paths to pruning but barriers between sparse solutions unless dense connectivity is allowed.

ABSTRACT

We investigate the difficulties of training sparse neural networks and make new observations about optimization dynamics and the energy landscape within the sparse regime. Recent work of \citep{Gale2019, Liu2018} has shown that sparse ResNet-50 architectures trained on ImageNet-2012 dataset converge to solutions that are significantly worse than those found by pruning. We show that, despite the failure of optimizers, there is a linear path with a monotonically decreasing objective from the initialization to the "good" solution. Additionally, our attempts to find a decreasing objective path from "bad" solutions to the "good" ones in the sparse subspace fail. However, if we allow the path to traverse the dense subspace, then we consistently find a path between two solutions. These findings suggest traversing extra dimensions may be needed to escape stationary points found in the sparse subspace.

Motivation & Objective

• Understand optimization dynamics and energy landscapes in sparse neural networks.
• Investigate why pruning-based methods outperform training sparse networks from scratch or with lottery initialization.
• Examine interpolation paths between initialization and sparse solutions to identify barriers in sparse subspaces.
• Assess whether removing sparsity constraints (allowing dense connectivity) helps escape local minima.
• Inform future methods for initialization, regularization, and optimization in sparsity-focused models.

Proposed method

• Train ResNet-50 on ImageNet-2012 under various sparsity regimes.
• Use magnitude-based pruning to reach predefined sparsity targets.
• Compare pruning-based solutions to scratch and lottery sparse initializations.
• Interpolate linearly and via Bézier curves between initialization and pruning solutions in both sparse and dense subspaces.
• Evaluate objective along interpolations to study energy landscapes and barriers.

Experimental results

Research questions

• RQ1 Why do sparse networks trained from scratch or with lottery initialization fail to match pruning results?
• RQ2 Is there a monotonically decreasing path from initialization to the pruned solution within the sparse subspace?
• RQ3 Do paths exist between sparse solutions when sparsity is relaxed to dense connectivity?
• RQ4 What does allowing dense connectivity imply for escaping bad stationary points in the sparse regime?

Key findings

• There exists a monotonically decreasing path from initialization to the pruned solution along linear interpolations within the sparse subspace.
• The linear path between scratch and pruned solutions shows a high energy barrier, and quadratic/cubic Bézier curves fail to provide a decreasing path in the sparse subspace.
• Removing the sparsity constraint (unmasking density) allows decreasing objective Bézier curves between two sparse solutions, suggesting dense connectivity helps escape stationary points.
• Paths in the dense subspace consistently find decreasing objective curves between two sparse solutions, indicating extra dimensions may be needed to escape sparse subspace stagnation.
• Pruned solutions remain difficult to reach from scratch or lottery initializations due to optimization dynamics in the sparse regime, despite monotonic paths to pruning.