[Paper Review] HENet:A Highly Efficient Convolutional Neural Networks Optimized for Accuracy, Speed and Storage

Qiuyu Zhu, Ruixin Zhang|arXiv (Cornell University)|Mar 7, 2018

Advanced Neural Network Applications15 references3 citations

TL;DR

HENet proposes a highly efficient convolutional neural network architecture that optimizes for accuracy, speed, and model size by combining group convolution with channel shuffling, integrating residual and dense connections within blocks, and eliminating pooling layers to preserve feature maps. Experiments show HENet achieves over 100% higher efficiency than ShuffleNet on CIFAR-10/100 and SVHN while maintaining competitive accuracy.

ABSTRACT

In order to enhance the real-time performance of convolutional neural networks(CNNs), more and more researchers are focusing on improving the efficiency of CNN. Based on the analysis of some CNN architectures, such as ResNet, DenseNet, ShuffleNet and so on, we combined their advantages and proposed a very efficient model called Highly Efficient Networks(HENet). The new architecture uses an unusual way to combine group convolution and channel shuffle which was mentioned in ShuffleNet. Inspired by ResNet and DenseNet, we also proposed a new way to use element-wise addition and concatenation connection with each block. In order to make greater use of feature maps, pooling operations are removed from HENet. The experiments show that our model's efficiency is more than 1 times higher than ShuffleNet on many open source datasets, such as CIFAR-10/100 and SVHN.

Motivation & Objective

• To address the growing demand for real-time inference in CNNs by improving efficiency across accuracy, speed, and model size.
• To overcome limitations in existing lightweight models like ShuffleNet, ResNet, and DenseNet by integrating their strengths into a unified architecture.
• To eliminate pooling operations to preserve spatial feature information and enhance feature map utilization.
• To develop a model that maintains high performance while significantly reducing computational and storage costs.

Proposed method

• Introduces a novel block design that combines element-wise addition (residual-style) and concatenation (dense-style) connections within each block to enhance feature reuse and gradient flow.
• Employs group convolution with channel shuffling to reduce computation and parameter count while maintaining channel cross-information.
• Removes pooling layers entirely to preserve spatial resolution and maximize feature map utilization across the network.
• Uses a hybrid skip connection strategy inspired by both ResNet and DenseNet to improve representational capacity without increasing computational overhead.
• Optimizes the network architecture through architectural analysis of existing models, focusing on efficiency trade-offs.
• Designs the network with minimal redundancy and high parameter efficiency to reduce storage and inference time.

Experimental results

Research questions

• RQ1 How can group convolution and channel shuffling be effectively combined to improve model efficiency without sacrificing accuracy?
• RQ2 To what extent can the integration of residual and dense connections enhance feature learning in a lightweight CNN?
• RQ3 What impact does removing pooling layers have on feature representation and overall model performance in small-scale datasets?
• RQ4 Can a unified architecture outperform specialized lightweight models like ShuffleNet in terms of speed, accuracy, and model size?
• RQ5 How does the proposed architecture compare to state-of-the-art models in terms of FLOPs, parameters, and inference speed on standard benchmarks?

Key findings

• HENet achieves over 100% higher efficiency than ShuffleNet on CIFAR-10, CIFAR-100, and SVHN datasets, measured by the ratio of accuracy to FLOPs.
• The model maintains competitive top-1 accuracy on CIFAR-10 and CIFAR-100, outperforming or matching state-of-the-art lightweight models despite reduced complexity.
• Removal of pooling layers leads to better preservation of spatial feature maps, contributing to improved performance on small-scale image recognition tasks.
• The integration of both residual and dense connections within blocks enhances feature propagation and gradient flow, improving training stability and accuracy.
• HENet reduces model size and FLOPs significantly compared to baseline models, making it suitable for deployment on resource-constrained devices.
• The architecture demonstrates strong generalization across diverse datasets, confirming its robustness and scalability.