[Paper Review] Slimmable Neural Networks

Jiahui Yu, Linjie Yang+3 more|arXiv (Cornell University)|Dec 21, 2018

Advanced Neural Network ApplicationsComputer Science38 references235 citations

TL;DR

A single neural network can run at multiple widths via switchable batch normalization, enabling on-device, runtime-adaptive accuracy–latency trade-offs with competitive performance to individually trained models across classification and detection tasks.

ABSTRACT

We present a simple and general method to train a single neural network executable at different widths (number of channels in a layer), permitting instant and adaptive accuracy-efficiency trade-offs at runtime. Instead of training individual networks with different width configurations, we train a shared network with switchable batch normalization. At runtime, the network can adjust its width on the fly according to on-device benchmarks and resource constraints, rather than downloading and offloading different models. Our trained networks, named slimmable neural networks, achieve similar (and in many cases better) ImageNet classification accuracy than individually trained models of MobileNet v1, MobileNet v2, ShuffleNet and ResNet-50 at different widths respectively. We also demonstrate better performance of slimmable models compared with individual ones across a wide range of applications including COCO bounding-box object detection, instance segmentation and person keypoint detection without tuning hyper-parameters. Lastly we visualize and discuss the learned features of slimmable networks. Code and models are available at: https://github.com/JiahuiYu/slimmable_networks

Motivation & Objective

• Motivate need for runtime-adaptive accuracy and latency on diverse devices.
• Propose a general framework to train a single network executable at multiple widths.
• Introduce Switchable Batch Normalization to stabilize multi-width training.
• Demonstrate competitive accuracy across ImageNet and downstream tasks (detection, segmentation, keypoints).

Proposed method

• Train a shared network across predefined width switches (e.g., [0.25,0.5,0.75,1.0]x).
• privatize Batch Normalization per switch (Switchable BN) to align statistics across widths.
• Train by aggregating losses across all switches in a single objective.
• Provide a memory-efficient training procedure that cycles through switches during each batch.
• Show that after training, switching width incurs negligible runtime/memory overhead.
• Demonstrate applicability to convolutions (including depthwise, group), pooling, FC, residuals, and various tasks.

Experimental results

Research questions

• RQ1 Can a single network be trained to operate effectively at multiple channel widths without significant accuracy loss?
• RQ2 Does Switchable Batch Normalization stabilize training and preserve performance across switches?
• RQ3 How does a slimmable network perform relative to individually trained networks at matching widths on ImageNet and downstream tasks?

Key findings

• Slimmable networks achieve similar (and often better) top-1 accuracy compared with individually trained networks at corresponding widths on ImageNet (MobileNet v1/v2, ShuffleNet, ResNet-50).
• Switchable Batch Normalization enables stable, rank-preserved accuracy across all switches during training and testing.
• Increasing the number of switches (e.g., from 4 to 8) maintains comparable performance and demonstrates scalability.
• Slimmable networks outperform or match individually trained models on COCO object detection, instance segmentation, and keypoint detection when evaluated at corresponding backbones/widths.
• Training with Switchable BN yields only negligible parameter overhead (BN layers typically <1% of model size) and can be fused at inference with no extra cost.
• The approach generalizes across standard and specialized convolutional blocks (including depthwise-separable and group convolutions) and across diverse vision tasks.