[Paper Review] Topological Insights into Sparse Neural Networks

Shiwei Liu, Tim van der Lee+6 more|TU/e Research Portal|Jun 24, 2020

Advanced Graph Neural Networks35 references4 citations

TL;DR

This paper introduces Neural Network Sparse Topology Distance (NNSTD), a graph-theory-based metric to measure topological differences between sparse neural networks. By applying NNSTD to analyze adaptive sparse connectivity, the authors demonstrate that many structurally distinct sparse subnetworks can achieve high performance—complementing the Lottery Ticket Hypothesis by showing that adaptive sparsity enables discovery of numerous high-performing, topologically diverse subnetworks without pre-training.

ABSTRACT

Sparse neural networks are effective approaches to reduce the resource requirements for the deployment of deep neural networks. Recently, the concept of adaptive sparse connectivity, has emerged to allow training sparse neural networks from scratch by optimizing the sparse structure during training. However, comparing different sparse topologies and determining how sparse topologies evolve during training, especially for the situation in which the sparse structure optimization is involved, remain as challenging open questions. This comparison becomes increasingly complex as the number of possible topological comparisons increases exponentially with the size of networks. In this work, we introduce an approach to understand and compare sparse neural network topologies from the perspective of graph theory. We first propose Neural Network Sparse Topology Distance (NNSTD) to measure the distance between different sparse neural networks. Further, we demonstrate that sparse neural networks can outperform over-parameterized models in terms of performance, even without any further structure optimization. To the end, we also show that adaptive sparse connectivity can always unveil a plenitude of sparse sub-networks with very different topologies which outperform the dense model, by quantifying and comparing their topological evolutionary processes. The latter findings complement the Lottery Ticket Hypothesis by showing that there is a much more efficient and robust way to find "winning tickets". Altogether, our results start enabling a better theoretical understanding of sparse neural networks, and demonstrate the utility of using graph theory to analyze them.

Motivation & Objective

• To address the lack of theoretical understanding in comparing sparse neural network topologies.
• To quantify how sparse topologies evolve during training, especially under adaptive sparse connectivity.
• To investigate whether multiple structurally different sparse subnetworks can achieve similar performance.
• To determine if adaptive sparse connectivity consistently discovers high-performing topologies across different initializations.
• To provide a foundation for theoretical analysis of sparse neural networks using graph-based metrics.

Proposed method

• Propose NNSTD, a novel distance metric inspired by Graph Edit Distance (GED), adapted to handle interchangeable hidden neurons in neural graphs.
• Model sparse neural networks as large neural graphs, treating weight connections as edges and neurons as nodes.
• Incorporate node permutation invariance into NNSTD to ensure topological similarity is preserved despite neuron reordering.
• Apply NNSTD to compare sparse topologies evolved via Sparse Evolutionary Training (SET) across multiple random seeds and pruning rates.
• Use retraining experiments with fixed topologies to isolate the contribution of topology vs. weight optimization in performance.
• Visualize topological evolution paths during training to analyze how sparse structures adapt over time.

Experimental results

Research questions

• RQ1 How different are high-performing sparse neural network topologies in terms of structure?
• RQ2 Can very different sparse topologies achieve similar performance, even without pre-training?
• RQ3 To what extent does adaptive sparse connectivity uncover diverse yet effective topologies?
• RQ4 How does the topological structure of a sparse network evolve during training with adaptive connectivity?
• RQ5 Is the performance of sparse networks due to topology, weight optimization, or both?

Key findings

• NNSTD successfully measures topological differences between sparse neural networks, enabling quantitative comparison of their structural diversity.
• Many structurally distinct sparse subnetworks achieve comparable or even better test accuracy than dense models, demonstrating the existence of multiple high-performing topologies.
• Adaptive sparse connectivity via SET consistently discovers a wide variety of topologically different subnetworks that all outperform the dense baseline.
• Retraining networks with fixed topologies from random weights shows performance improves over time, indicating topology evolution is critical for success.
• The combination of weight and topology optimization is essential—randomly initialized sparse networks with optimized topologies outperform those with only weight optimization.
• The results suggest that adaptive sparse connectivity is a robust method to discover high-performing subnetworks without relying on lucky initialization or pre-training.