[Paper Review] PHEW: Constructing Sparse Networks that Learn Fast and Generalize Well without Training Data

Shreyas Malakarjun Patil, Constantine Dovrolis|arXiv (Cornell University)|Oct 22, 2020

Machine Learning and ELM4 citations

TL;DR

This paper proposes PHEW, a data-agnostic method to construct sparse neural networks with fast convergence and strong generalization by selecting input-output paths based on initial weights via biased random walks. PHEW achieves state-of-the-art performance across multiple architectures and datasets without training data, outperforming SynFlow-L2 and other pruning-at-initialization methods by maintaining wider layers while preserving high path kernel trace.

ABSTRACT

Methods that sparsify a network at initialization are important in practice because they greatly improve the efficiency of both learning and inference. Our work is based on a recently proposed decomposition of the Neural Tangent Kernel (NTK) that has decoupled the dynamics of the training process into a data-dependent component and an architecture-dependent kernel - the latter referred to as Path Kernel. That work has shown how to design sparse neural networks for faster convergence, without any training data, using the Synflow-L2 algorithm. We first show that even though Synflow-L2 is optimal in terms of convergence, for a given network density, it results in sub-networks with "bottleneck" (narrow) layers - leading to poor performance as compared to other data-agnostic methods that use the same number of parameters. Then we propose a new method to construct sparse networks, without any training data, referred to as Paths with Higher-Edge Weights (PHEW). PHEW is a probabilistic network formation method based on biased random walks that only depends on the initial weights. It has similar path kernel properties as Synflow-L2 but it generates much wider layers, resulting in better generalization and performance. PHEW achieves significant improvements over the data-independent SynFlow and SynFlow-L2 methods at a wide range of network densities.

Motivation & Objective

• To address the poor generalization of data-agnostic pruning methods that produce narrow bottleneck layers despite high convergence speed.
• To develop a method that maximizes path kernel trace for fast training while maintaining wide, well-distributed layer widths for better generalization.
• To enable sparse network construction without any training data, relying only on initial network weights and probabilistic path selection.
• To improve upon SynFlow-L2 by avoiding layer-collapse and suboptimal architectural structures while preserving convergence speed.
• To validate that path-based sparsification with weight-biased random walks yields superior performance across diverse architectures and weight initializations.

Proposed method

• PHEW constructs sparse networks by selecting input-output paths based on initial weight magnitudes, avoiding pruning individual connections or units.
• It uses biased random walks that favor higher-weight edges at each layer, ensuring high path kernel trace for fast convergence.
• The method preserves per-layer width by conserving entire paths rather than individual neurons or edges, preventing bottleneck layers.
• Sparsification is performed solely on the untrained, initialized network using only the initial weight values, with no gradient computation or data access.
• The algorithm is probabilistic: at each layer, the next hop in the path is selected with probability proportional to the magnitude of the incoming weights.
• PHEW is robust across different weight initialization schemes (Kaiming, Xavier, Normal), as shown empirically on VGG19 and ResNet20.

Experimental results

Research questions

• RQ1 Can a data-agnostic pruning method achieve faster convergence and better generalization than SynFlow-L2 by avoiding bottleneck layers?
• RQ2 Does selecting paths based on initial weights via biased random walks yield higher path kernel trace and better performance than uniform or inverse-weight random walks?
• RQ3 How does PHEW’s performance compare to random pruning and reinitialized sparse networks when using the same architecture but different weight initializations?
• RQ4 Can the sparse architecture produced by PHEW be decoupled from the initial weights, such that reinitializing the weights does not degrade performance?
• RQ5 Is it possible to construct sparse networks with high path kernel trace and wide layers without using any training data?

Key findings

• PHEW achieves significant performance improvements over SynFlow-L2 and other data-agnostic pruning methods across CIFAR-10, CIFAR-100, VGG19, and ResNet20 at various network densities.
• PHEW maintains high per-layer width, avoiding the bottleneck layers common in SynFlow-L2, which leads to better generalization despite similar path kernel trace.
• The path kernel trace of PHEW is close to that of SynFlow-L2, indicating comparable convergence speed, while generalization performance is substantially better.
• PHEW’s performance is robust across different weight initialization methods (Kaiming, Xavier, Normal), indicating insensitivity to initialization distribution.
• Reinitialized PHEW networks (with new weights but same architecture) achieve performance comparable to original PHEW, confirming that the architecture is the key factor, not the initial weights.
• PHEW outperforms both unbiased and inverse-weight-biased random walk variants, which have lower path kernel trace and slower initial convergence.