[Paper Review] AttendNets: Tiny Deep Image Recognition Neural Networks for the Edge via Visual Attention Condensers

Alexander Wong, Mahmoud Famouri+1 more|arXiv (Cornell University)|Sep 29, 2020

Advanced Memory and Neural ComputingEngineering17 citations

TL;DR

AttendNets introduces highly efficient, low-precision deep neural networks for on-device image recognition using visual attention condensers—self-attention mechanisms that enhance spatial-channel selective attention—combined with machine-driven architecture search. The approach achieves state-of-the-art accuracy with up to 16.7× lower memory and 3× fewer FLOPs than MobileNet-V1, demonstrating a superior accuracy-complexity trade-off for TinyML applications.

ABSTRACT

While significant advances in deep learning has resulted in state-of-the-art performance across a large number of complex visual perception tasks, the widespread deployment of deep neural networks for TinyML applications involving on-device, low-power image recognition remains a big challenge given the complexity of deep neural networks. In this study, we introduce AttendNets, low-precision, highly compact deep neural networks tailored for on-device image recognition. More specifically, AttendNets possess deep self-attention architectures based on visual attention condensers, which extends on the recently introduced stand-alone attention condensers to improve spatial-channel selective attention. Furthermore, AttendNets have unique machine-designed macroarchitecture and microarchitecture designs achieved via a machine-driven design exploration strategy. Experimental results on ImageNet$_{50}$ benchmark dataset for the task of on-device image recognition showed that AttendNets have significantly lower architectural and computational complexity when compared to several deep neural networks in research literature designed for efficiency while achieving highest accuracies (with the smallest AttendNet achieving $\\sim$7.2% higher accuracy, while requiring $\\sim$3$\ imes$ fewer multiply-add operations, $\\sim$4.17$\ imes$ fewer parameters, and $\\sim$16.7$\ imes$ lower weight memory requirements than MobileNet-V1). Based on these promising results, AttendNets illustrate the effectiveness of visual attention condensers as building blocks for enabling various on-device visual perception tasks for TinyML applications.

Motivation & Objective

• To address the challenge of deploying complex deep neural networks on low-power, on-device edge systems for image recognition.
• To improve efficiency and accuracy in TinyML by introducing a novel self-attention mechanism tailored for edge deployment.
• To leverage machine-driven design exploration for optimizing both macro- and micro-architectures of compact neural networks.
• To demonstrate that visual attention condensers can outperform existing efficient architectures in accuracy and computational efficiency.

Proposed method

• Visual attention condensers (VACs) are introduced as self-attention modules that downmix, condense, embed, expand, and selectively attend to spatial-channel activations for improved feature representation.
• The VAC architecture includes a down-mixing layer to reduce channel dimensions, a condensation layer to focus on strong activation proximity, and an expansion layer to reconstruct attention maps.
• A selective attention mechanism fuses the original features with learned attention values using scale modulation, enabling spatial-channel selective refinement.
• The AttendNet architecture is generated via a machine-driven design exploration strategy that automatically determines optimal macro- and micro-architecture configurations.
• Low-precision quantization is applied to reduce model size and computational cost, enabling deployment on resource-constrained edge devices.
• The method integrates attention-based feature refinement with efficient operations such as pointwise group convolutions and channel shuffling to reduce FLOPs and parameters.

Experimental results

Research questions

• RQ1 Can visual attention condensers significantly improve the accuracy and efficiency of compact deep neural networks for on-device image recognition?
• RQ2 How does a machine-driven design exploration strategy optimize the macro- and micro-architecture of tiny attention-based networks?
• RQ3 To what extent can self-attention mechanisms outperform traditional efficiency-focused architectures like MobileNet-V1 and MobileNet-V2 in accuracy and complexity?
• RQ4 How do visual attention condensers compare to existing attention mechanisms in terms of spatial-channel selectivity and computational efficiency?

Key findings

• AttendNet-A achieved ~8.7% higher top-1 accuracy than MobileNet-V1 while requiring ~2.35× fewer parameters, ~9.4× lower weight memory, and ~2.1× fewer multiply-add operations.
• The smallest AttendNet achieved ~7.2% higher accuracy than MobileNet-V1 with ~3× fewer FLOPs, ~4.17× fewer parameters, and ~16.7× lower weight memory.
• AttendNet-A outperformed AttoNet-A by ~0.2% top-1 accuracy while using ~2.1× fewer parameters, ~8.4× lower memory, and ~1.53× fewer FLOPs.
• The integration of visual attention condensers and machine-driven design led to a superior balance between accuracy and model complexity across all evaluated metrics.
• AttendNets demonstrated strong generalization and efficiency, making them suitable for real-time, low-power edge inference tasks.