[Paper Review] Splitting Steepest Descent for Growing Neural Architectures

Qiang Liu, Lemeng Wu+1 more|arXiv (Cornell University)|Oct 6, 2019

Human Pose and Action Recognition29 references20 citations

TL;DR

This paper proposes Splitting Steepest Descent (SSD), a novel progressive neural architecture search method that adaptively grows neural networks by splitting neurons into off-springs using a functional steepest descent framework in an ∞-Wasserstein metric space. The method achieves second-order descent for escaping saddle points, outperforms random and heuristic splitting strategies, and enables efficient, lightweight architecture learning in resource-constrained settings.

ABSTRACT

We develop a progressive training approach for neural networks which adaptively grows the network structure by splitting existing neurons to multiple off-springs. By leveraging a functional steepest descent idea, we derive a simple criterion for deciding the best subset of neurons to split and a splitting gradient for optimally updating the off-springs. Theoretically, our splitting strategy is a second-order functional steepest descent for escaping saddle points in an $\infty$-Wasserstein metric space, on which the standard parametric gradient descent is a first-order steepest descent. Our method provides a new computationally efficient approach for optimizing neural network structures, especially for learning lightweight neural architectures in resource-constrained settings.

Motivation & Objective

• To address the challenge of efficient, automatic neural architecture optimization in resource-constrained settings.
• To extend gradient-based optimization beyond parameters to the discrete space of model structures.
• To develop a principled, differentiable method for growing neural networks by splitting neurons rather than random initialization or fixed additions.
• To enable faster convergence and better performance in continual learning and lightweight model learning.
• To provide a second-order functional steepest descent approach that escapes saddle points via optimal splitting.

Proposed method

• The method alternates between standard parametric gradient descent and a splitting phase that grows the network by splitting neurons into off-springs.
• Splitting is guided by a functional steepest descent criterion in an ∞-Wasserstein metric space, treating the network as a distribution of neurons.
• The optimal splitting direction is derived from the splitting gradient, which minimizes the loss under infinitesimal perturbations of the off-springs.
• The splitting steepest descent is theoretically shown to be a second-order descent, while standard parametric descent is first-order.
• The method uses a kernel-based MMD (Maximum Mean Discrepancy) loss to measure distributional similarity between compressed and original data distributions.
• The splitting matrix is computed via second-order derivatives of the kernel function, enabling optimal off-spring placement.

Experimental results

Research questions

• RQ1 Can we derive a principled, differentiable strategy for growing neural network architectures by splitting neurons?
• RQ2 How can functional steepest descent in an ∞-Wasserstein space be used to guide optimal neuron splitting?
• RQ3 Does splitting-based architecture growth outperform random or heuristic splitting in convergence and accuracy?
• RQ4 Can this method escape saddle points more effectively than standard parametric gradient descent?
• RQ5 How does the proposed method compare to Frank-Wolfe, random initialization, and pruning-based approaches in learning lightweight models?

Key findings

• The proposed Splitting Steepest Descent (Optimal Split) achieves the lowest MMD loss across all iterations in the data compression experiment, outperforming all baselines.
• The method converges faster and reaches lower training loss than Random Split, New Initialization, and Gradient Boosting, especially in early iterations.
• The kicks in loss curves for New Initialization and Gradient Boosting are attributed to re-weighting after each new particle is added, which the proposed method avoids.
• The method enables effective model growth without requiring large pre-trained models, making it ideal for resource-constrained settings like mobile and IoT devices.
• Empirical results show that SSD can learn more accurate small models than existing pruning methods, demonstrating its superiority in compact architecture learning.
• Theoretical analysis confirms that the splitting phase performs a second-order functional steepest descent, enabling escape from saddle points in the functional space.