[Paper Review] Binarized Neural Architecture Search for Efficient Object Recognition

Hanlin Chen, Li’an Zhuo+6 more|arXiv (Cornell University)|Sep 8, 2020

Advanced Neural Network Applications73 references4 citations

TL;DR

This paper proposes Binarized Neural Architecture Search (BNAS), a method that searches for binarized convolutional neural network architectures to enable efficient object recognition on edge devices. By integrating operation space reduction, channel sampling, and a performance-based pruning strategy with Upper Confidence Bound (UCB) optimization, BNAS achieves comparable accuracy to full-precision NAS methods—96.53% on CIFAR-10 and strong performance on ImageNet and face recognition—while reducing model size and search time by up to 40% compared to PC-DARTS.

ABSTRACT

Traditional neural architecture search (NAS) has a significant impact in computer vision by automatically designing network architectures for various tasks. In this paper, binarized neural architecture search (BNAS), with a search space of binarized convolutions, is introduced to produce extremely compressed models to reduce huge computational cost on embedded devices for edge computing. The BNAS calculation is more challenging than NAS due to the learning inefficiency caused by optimization requirements and the huge architecture space, and the performance loss when handling the wild data in various computing applications. To address these issues, we introduce operation space reduction and channel sampling into BNAS to significantly reduce the cost of searching. This is accomplished through a performance-based strategy that is robust to wild data, which is further used to abandon less potential operations. Furthermore, we introduce the Upper Confidence Bound (UCB) to solve 1-bit BNAS. Two optimization methods for binarized neural networks are used to validate the effectiveness of our BNAS. Extensive experiments demonstrate that the proposed BNAS achieves a comparable performance to NAS on both CIFAR and ImageNet databases. An accuracy of $96.53\%$ vs. $97.22\%$ is achieved on the CIFAR-10 dataset, but with a significantly compressed model, and a $40\%$ faster search than the state-of-the-art PC-DARTS. On the wild face recognition task, our binarized models achieve a performance similar to their corresponding full-precision models.

Motivation & Objective

• To address the high computational cost and large model size of neural networks on edge devices.
• To develop a neural architecture search method tailored for binarized networks, which are more efficient but suffer from performance degradation.
• To reduce the search cost and improve efficiency in searching binarized architectures, especially for 1-bit networks.
• To enable robust performance on wild data, such as real-world face recognition, with minimal accuracy drop.
• To demonstrate that binarized architectures can match full-precision NAS performance while being significantly more compressed.

Proposed method

• Introduces BNAS, a search space composed of binarized convolutions to generate highly compressed models for edge deployment.
• Applies operation space reduction and channel sampling to prune the search space early, reducing computational cost.
• Employs a performance-based strategy to dynamically eliminate low-potential operations during search, improving efficiency and robustness to wild data.
• Uses the Upper Confidence Bound (UCB) bandit algorithm to optimize 1-bit BNAS, balancing exploration and exploitation in architecture search.
• Reformulates binarization optimization within the same NAS framework, enabling end-to-end training and search.
• Validates the method using two binarized network optimization techniques: PCNN and XNOR-Net, demonstrating generalization across methods.

Experimental results

Research questions

• RQ1 Can neural architecture search be effectively adapted to binarized networks to achieve high efficiency and performance?
• RQ2 How can the search space be reduced efficiently in binarized NAS without sacrificing accuracy?
• RQ3 Can a performance-based pruning strategy improve robustness to wild data in binarized models?
• RQ4 How does the UCB bandit strategy enhance 1-bit BNAS performance compared to random or greedy search?
• RQ5 To what extent can binarized architectures match full-precision NAS performance on real-world tasks like face recognition?

Key findings

• BNAS achieves 96.53% top-1 accuracy on CIFAR-10, compared to 97.22% for full-precision NAS, with a significantly compressed model.
• The search time for BNAS is 0.09375 GPU days, which is 40% faster than the state-of-the-art PC-DARTS method.
• On the ImageNet dataset, BNAS discovers binarized architectures that perform comparably to full-precision NAS models, with much smaller model size.
• For face recognition, 1-bit BNAS achieves 98.57% accuracy on LFW, only 1.06% lower than full-precision ResNet-18, while using just 10.224M parameters.
• 1-bit BNAS outperforms existing human-designed 1-bit networks by 5.0% in accuracy, with fewer parameters.
• The method demonstrates strong transferability, maintaining high performance across diverse datasets including CIFAR-10, ImageNet, and wild face recognition benchmarks.