[Paper Review] Finding trainable sparse networks through Neural Tangent Transfer

Tianlin Liu, Friedemann Zenke|arXiv (Cornell University)|Jun 15, 2020

Neural Networks and Applications35 references4 citations

TL;DR

This paper introduces Neural Tangent Transfer (NTT), a label-free foresight pruning method that identifies trainable sparse neural networks by matching their training dynamics—characterized by the Neural Tangent Kernel (NTK)—to those of dense networks. The approach enables faster convergence and higher accuracy in sparse convolutional networks without relying on labeled data, particularly excelling at sparsifying convolutional layers.

ABSTRACT

Deep neural networks have dramatically transformed machine learning, but their memory and energy demands are substantial. The requirements of real biological neural networks are rather modest in comparison, and one feature that might underlie this austerity is their sparse connectivity. In deep learning, trainable sparse networks that perform well on a specific task are usually constructed using label-dependent pruning criteria. In this article, we introduce Neural Tangent Transfer, a method that instead finds trainable sparse networks in a label-free manner. Specifically, we find sparse networks whose training dynamics, as characterized by the neural tangent kernel, mimic those of dense networks in function space. Finally, we evaluate our label-agnostic approach on several standard classification tasks and show that the resulting sparse networks achieve higher classification performance while converging faster.

Motivation & Objective

• To develop a label-free method for identifying trainable sparse neural networks that maintain training dynamics similar to dense networks.
• To overcome limitations of existing foresight pruning methods that rely on labeled data or global pruning criteria.
• To enable effective layerwise sparsification of convolutional filters, which are computationally expensive, for energy-efficient inference.
• To leverage the Neural Tangent Kernel (NTK) to ensure sparse networks generalize well and train rapidly.
• To provide a theoretically grounded, initialization-based approach that transfers training dynamics from dense to sparse networks.

Proposed method

• The method uses the Neural Tangent Kernel (NTK) to compare the function-space training dynamics of sparse and dense networks during initialization.
• It formulates a pruning criterion based on minimizing the deviation of the sparse network's NTK from that of the dense network in the function space.
• The approach is label-agnostic, relying only on unlabeled data to compute NTK matrices and guide pruning.
• NTT performs layerwise pruning, specifically targeting convolutional filters, to maximize computational efficiency.
• The method identifies sparse network structures that preserve the training trajectory of the dense network, ensuring fast convergence and high accuracy.
• It leverages theoretical insights from linearized neural networks and NTK theory to ensure stable and generalizable training dynamics.

Experimental results

Research questions

• RQ1 Can a label-free method identify trainable sparse neural networks that match the training dynamics of dense networks?
• RQ2 Can NTT effectively sparsify convolutional filters in a layerwise manner, improving energy efficiency in CNNs?
• RQ3 Does NTT outperform existing foresight pruning methods in terms of convergence speed and final accuracy?
• RQ4 How does NTT's performance vary across different sparsity levels and architectures?
• RQ5 Can NTK-based training dynamics transfer from dense to sparse networks without requiring supervised signals?

Key findings

• NTT successfully identifies trainable sparse networks without using any labeled data, making it suitable for data-scarce scenarios.
• The method achieves higher classification accuracy than baseline pruning methods on CIFAR-10 and MNIST benchmarks.
• NTT enables faster convergence, with sparse networks trained to high accuracy in fewer epochs compared to standard pruning baselines.
• Layerwise pruning via NTT significantly reduces computational cost by sparsifying convolutional filters, which are the main source of FLOPs in CNNs.
• Theoretical speedups of up to 4x are achieved at 1% sparsity, with minimal accuracy drop, demonstrating strong efficiency-accuracy tradeoffs.
• NTT outperforms existing foresight pruning methods in preserving training dynamics, especially in convolutional layers where global pruning fails.