[Paper Review] AutoSlim: Towards One-Shot Architecture Search for Channel Numbers

Jiahui Yu, Thomas S. Huang|arXiv (Cornell University)|Mar 27, 2019

Advanced Neural Network Applications60 references145 citations

TL;DR

AutoSlim trains a slimmable network to estimate accuracy across channel configurations and greedily slim layers to meet resource constraints, yielding optimized channel numbers without exhaustive search. It achieves better speed-accuracy trade-offs than baselines on ImageNet across MobileNet variants and ResNet-50.

ABSTRACT

We study how to set channel numbers in a neural network to achieve better accuracy under constrained resources (e.g., FLOPs, latency, memory footprint or model size). A simple and one-shot solution, named AutoSlim, is presented. Instead of training many network samples and searching with reinforcement learning, we train a single slimmable network to approximate the network accuracy of different channel configurations. We then iteratively evaluate the trained slimmable model and greedily slim the layer with minimal accuracy drop. By this single pass, we can obtain the optimized channel configurations under different resource constraints. We present experiments with MobileNet v1, MobileNet v2, ResNet-50 and RL-searched MNasNet on ImageNet classification. We show significant improvements over their default channel configurations. We also achieve better accuracy than recent channel pruning methods and neural architecture search methods. Notably, by setting optimized channel numbers, our AutoSlim-MobileNet-v2 at 305M FLOPs achieves 74.2% top-1 accuracy, 2.4% better than default MobileNet-v2 (301M FLOPs), and even 0.2% better than RL-searched MNasNet (317M FLOPs). Our AutoSlim-ResNet-50 at 570M FLOPs, without depthwise convolutions, achieves 1.3% better accuracy than MobileNet-v1 (569M FLOPs). Code and models will be available at: https://github.com/JiahuiYu/slimmable_networks

Motivation & Objective

• Motivate and address how to set per-layer channel numbers to improve accuracy under constrained resources.
• Propose a one-shot method to estimate and optimize channel configurations without training many architectures.
• Show that slimmable networks can serve as accurate estimators for relative performance across channel configurations.
• Demonstrate state-of-the-art accuracy-efficiency trade-offs on ImageNet for MobileNet variants, MNasNet, and ResNet-50 under various FLOPs budgets.

Proposed method

• Train a slimmable network that can run at arbitrary widths using sandwich rule and inplace distillation.
• Greedily Slim layers by evaluating validation accuracy after slimming each layer’s group of channels and removing the one with the smallest accuracy drop.
• Iterate until meeting a specified resource constraint (FLOPs, latency, memory, or model size).
• Use the trained slimmable model to sample sub-architectures for instant inference and relative performance estimation.
• After search, train the resulting optimized architectures from scratch (or jointly as a slimmable network) for full epochs.
• Compare against default channel configurations, channel pruning baselines, and NAS-based methods on ImageNet.

Experimental results

Research questions

• RQ1 Can a single slimmable network accurately estimate the relative performance of many channel configurations?
• RQ2 Can a greedy slimming procedure on a trained slimmable model yield near-optimal channel configurations under given resource constraints?
• RQ3 Do optimized channel configurations improve accuracy and efficiency beyond standard heuristics, pruning, or prior NAS results?
• RQ4 How do AutoSlim-derived configurations perform across MobileNet variants, MNasNet, and ResNet-50 under different FLOPs budgets?

Key findings

• AutoSlim improves top-1 accuracy over default channel configurations across MobileNet v1/v2, MNasNet, and ResNet-50 under matched FLOPs budgets.
• AutoSlim-MobileNet-v2 at 305M FLOPs achieves 74.2% top-1, 2.4% above default MobileNet-v2 and 0.2% above RL-searched MNasNet at 317M FLOPs.
• AutoSlim-ResNet-50 at 570M FLOPs (no depthwise convs) achieves 1.3% higher accuracy than MobileNet-v1 at 569M FLOPs.
• The optimized channel configurations tend to reduce shallow-layer channels and increase deeper-layer channels compared to defaults, as seen in MobileNet v2 analyses.
• AutoSlim can outperform recent channel pruning and NAS baselines on ImageNet classifications under comparable computational budgets.