[Paper Review] STEERAGE: Synthesis of Neural Networks Using Architecture Search and Grow-and-Prune Methods

Shayan Hassantabar, Xiaoliang Dai+1 more|arXiv (Cornell University)|Dec 12, 2019

Advanced Neural Network Applications50 references20 citations

TL;DR

STEERAGE proposes a two-stage neural network synthesis framework combining efficient architecture search with grow-and-prune optimization to generate accurate, compact models. It uses a boosted decision tree-based accuracy predictor to accelerate search and applies targeted growth and pruning to enhance performance and reduce redundancy, achieving state-of-the-art accuracy on CIFAR-10 with 3.86% error rate using only 0.41M parameters in a 40-layer ResNet variant.

ABSTRACT

Neural networks (NNs) have been successfully deployed in many applications. However, architectural design of these models is still a challenging problem. Moreover, neural networks are known to have a lot of redundancy. This increases the computational cost of inference and poses an obstacle to deployment on Internet-of-Thing sensors and edge devices. To address these challenges, we propose the STEERAGE synthesis methodology. It consists of two complementary approaches: efficient architecture search, and grow-and-prune NN synthesis. The first step, covered in a global search module, uses an accuracy predictor to efficiently navigate the architectural search space. The predictor is built using boosted decision tree regression, iterative sampling, and efficient evolutionary search. The second step involves local search. By using various grow-and-prune methodologies for synthesizing convolutional and feed-forward NNs, it reduces the network redundancy, while boosting its performance. We have evaluated STEERAGE performance on various datasets, including MNIST and CIFAR-10. On MNIST dataset, our CNN architecture achieves an error rate of 0.66%, with 8.6x fewer parameters compared to the LeNet-5 baseline. For the CIFAR-10 dataset, we used the ResNet architectures as the baseline. Our STEERAGE-synthesized ResNet-18 has a 2.52% accuracy improvement over the original ResNet-18, 1.74% over ResNet-101, and 0.16% over ResNet-1001, while having comparable number of parameters and FLOPs to the original ResNet-18. This shows that instead of just increasing the number of layers to increase accuracy, an alternative is to use a better NN architecture with fewer layers. In addition, STEERAGE achieves an error rate of just 3.86% with a variant of ResNet architecture with 40 layers. To the best of our knowledge, this is the highest accuracy obtained by ResNet-based architectures on the CIFAR-10 dataset.

Motivation & Objective

• Address the challenge of large, exponentially growing neural architecture search spaces in deep learning.
• Overcome the inefficiency and suboptimality of trial-and-error hyperparameter tuning in neural network design.
• Reduce model redundancy and computational cost to enable deployment on edge devices and IoT sensors.
• Simultaneously improve model accuracy and compactness without relying on deep, resource-heavy architectures.
• Develop a generalizable, efficient framework applicable to both feedforward and convolutional neural networks.

Proposed method

• Employ a global search module using a boosted decision tree regression model as an accuracy predictor to estimate architecture performance without full training.
• Integrate quasi Monte-Carlo (QMC) sampling and iterative sampling to efficiently train the accuracy predictor across the search space.
• Apply an efficient evolutionary search (EES) algorithm to explore and identify high-performing architecture variants from the base architecture.
• Use a local search phase with two distinct grow-and-prune methodologies: SCANN for feedforward networks and NeST for convolutional layers.
• For feedforward networks, apply iterative connection growth, neuron growth, and connection pruning to enhance accuracy and compactness.
• For convolutional layers, use feature map growth and partial-area convolution with configurable pruning ratios to balance accuracy and efficiency.

Experimental results

Research questions

• RQ1 Can an accuracy predictor based on boosted decision trees and iterative sampling significantly accelerate neural architecture search while maintaining high performance?
• RQ2 To what extent can grow-and-prune synthesis improve both accuracy and model compactness in neural networks?
• RQ3 Can a hybrid global+local search strategy outperform traditional NAS methods in terms of efficiency and final model quality?
• RQ4 Is it possible to achieve state-of-the-art accuracy on CIFAR-10 with a shallower, smaller ResNet variant compared to deeper, larger baselines?
• RQ5 How does the integration of dimensionality reduction and hyperparameter search over depth, width, and pruning ratios affect the final model’s performance and efficiency?

Key findings

• The STEERAGE-synthesized ResNet-40 achieved a test error rate of 3.86% on CIFAR-10, outperforming even the 1001-layer ResNet-1001, which had a 4.62% error rate.
• The ResNet-18 variant synthesized by STEERAGE achieved a 4.46% error rate, representing a 2.52% improvement over the original ResNet-18 and a 1.74% improvement over the much deeper ResNet-101.
• A compact variant of the ResNet-18 synthesized by STEERAGE achieved 4.78% error rate with 2.1× fewer parameters than the original ResNet-18.
• The accuracy predictor enabled a 6000× speedup in architecture evaluation, reducing inference time from ~2 GPU hours per model to 1.2 seconds.
• The framework successfully generated a 40-layer ResNet with only 0.41M parameters and 1.30G FLOPs, achieving the highest accuracy among all ResNet-based architectures on CIFAR-10.
• The combined global+local search approach consistently outperformed global search alone, demonstrating that local refinement enhances both accuracy and compactness.