[Paper Review] SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size

Forrest Iandola, Song Han+4 more|arXiv (Cornell University)|Feb 24, 2016

Anomaly Detection Techniques and Applications662 citations

TL;DR

SqueezeNet achieves AlexNet-level accuracy on ImageNet with 50x fewer parameters and compresses to under 0.5MB. The paper also analyzes micro- and macro-architectural design space for compact CNNs.

ABSTRACT

Recent research on deep neural networks has focused primarily on improving accuracy. For a given accuracy level, it is typically possible to identify multiple DNN architectures that achieve that accuracy level. With equivalent accuracy, smaller DNN architectures offer at least three advantages: (1) Smaller DNNs require less communication across servers during distributed training. (2) Smaller DNNs require less bandwidth to export a new model from the cloud to an autonomous car. (3) Smaller DNNs are more feasible to deploy on FPGAs and other hardware with limited memory. To provide all of these advantages, we propose a small DNN architecture called SqueezeNet. SqueezeNet achieves AlexNet-level accuracy on ImageNet with 50x fewer parameters. Additionally, with model compression techniques we are able to compress SqueezeNet to less than 0.5MB (510x smaller than AlexNet). The SqueezeNet architecture is available for download here: https://github.com/DeepScale/SqueezeNet

Motivation & Objective

• Identify CNN architectures with far fewer parameters that maintain AlexNet-level accuracy on ImageNet.
• Propose architectural building blocks (Fire module) and design strategies to reduce parameter count.
• Evaluate compression potential and understand micro/macro architectural impacts on size and accuracy.
• Explore design-space space to provide principled guidance for compact CNN design.

Proposed method

• Introduce Fire module as a squeeze-expand block with 1x1 squeeze layers feeding into expand layers (1x1 and 3x3).
• Apply design strategies: replace 3x3 with 1x1 filters, reduce input channels to 3x3 filters via squeeze layers, and delay downsampling to keep large activation maps.
• Construct SqueezeNet architecture with eight Fire modules and late pooling, and remove fully-connected layers.
• Train with ReLU activations, dropout after Fire9, and a specific learning-rate schedule; implement expand module as two parallel conv layers (1x1 and 3x3) concatenated.
• Evaluate against AlexNet baseline and prior compression methods, and compare model sizes and accuracies; apply Deep Compression to achieve ultra-small models.

Experimental results

Research questions

• RQ1 Can a CNN with dramatically fewer parameters match AlexNet-level accuracy on ImageNet?
• RQ2 What architectural choices (Fire module, squeeze ratio, 1x1 vs 3x3 mix, downsampling schedule) most reduce parameter count without sacrificing accuracy?
• RQ3 How does post-hoc compression (quantization/pruning) affect extremely compact CNNs like SqueezeNet?
• RQ4 What macroarchitectural modifications (bypass connections) impact accuracy and size in compact CNNs?

Key findings

• SqueezeNet achieves 50x reduction in parameters with AlexNet-level Top-1/Top-5 accuracy on ImageNet (57.5% / 80.3%).
• With Deep Compression (8-bit, 6-bit) SqueezeNet can be compressed to 0.66MB (363x) and 0.47MB (510x) while maintaining or exceeding baseline accuracy.
• Macroarchitectural bypass connections (simple/bypass) improve Top-1 by up to ~2.9 percentage points without increasing model size.
• Microarchitectural exploration shows higher Top-5 accuracy (86.0%) at 19MB when SR and 3x3 mix are adjusted; 4.8MB baseline is AlexNet-level.
• SqueezeNet model size becomes extremely favorable for FPGA/embedded deployment and over-the-air updates.