[Paper Review] GhostNetV2: Enhance Cheap Operation with Long-Range Attention

Yehui Tang, Kai Han+4 more|arXiv (Cornell University)|Nov 23, 2022

Advanced Neural Network ApplicationsComputer Science277 citations

TL;DR

GhostNetV2 introduces a hardware-friendly decoupled fully connected (DFC) attention mechanism to enhance GhostNet by capturing long-range dependencies, achieving 75.3% top-1 accuracy on ImageNet at 167 MFLOPs, outperforming GhostNetV1.

ABSTRACT

Light-weight convolutional neural networks (CNNs) are specially designed for applications on mobile devices with faster inference speed. The convolutional operation can only capture local information in a window region, which prevents performance from being further improved. Introducing self-attention into convolution can capture global information well, but it will largely encumber the actual speed. In this paper, we propose a hardware-friendly attention mechanism (dubbed DFC attention) and then present a new GhostNetV2 architecture for mobile applications. The proposed DFC attention is constructed based on fully-connected layers, which can not only execute fast on common hardware but also capture the dependence between long-range pixels. We further revisit the expressiveness bottleneck in previous GhostNet and propose to enhance expanded features produced by cheap operations with DFC attention, so that a GhostNetV2 block can aggregate local and long-range information simultaneously. Extensive experiments demonstrate the superiority of GhostNetV2 over existing architectures. For example, it achieves 75.3% top-1 accuracy on ImageNet with 167M FLOPs, significantly suppressing GhostNetV1 (74.5%) with a similar computational cost. The source code will be available at https://github.com/huawei-noah/Efficient-AI-Backbones/tree/master/ghostnetv2_pytorch and https://gitee.com/mindspore/models/tree/master/research/cv/ghostnetv2.

Motivation & Objective

• Motivate efficient image classification on mobile devices with strong accuracy.
• Improve expressiveness of light-weight backbones without sacrificing deployment speed.
• Incorporate long-range information while maintaining hardware-friendly computations.
• Enhance GhostNet by leveraging a decoupled attention mechanism to augment expanded features.

Proposed method

• Propose DFC attention based on fully-connected layers decomposed into horizontal and vertical components to produce global receptive fields.
• Down-sample features to half resolution to reduce DFC computation and up-sample to original size for final fusion.
• Embed DFC attention in GhostNet blocks by multiplying expanded Ghost features with a sigmoid-scaled attention map.
• Enhance the Ghost module’s expanded features rather than the output features to improve expressiveness.
• Implement DFC attention as two depth-wise convolutions (1xKH and KWx1) for efficient hardware-friendly computation.

Experimental results

Research questions

• RQ1 Can a hardware-friendly long-range attention mechanism improve performance of light-weight CNNs on mobile devices?
• RQ2 Does enhancing the expanded (expressive) features of GhostNet via DFC attention yield greater gains than enhancing the output features?
• RQ3 What is the trade-off between accuracy and practical latency when integrating DFC attention into GhostNet?
• RQ4 How does GhostNetV2 perform across ImageNet, COCO, and ADE20K compared to prior light-weight backbones?

Key findings

• GhostNetV2 achieves 75.3% top-1 accuracy on ImageNet at 167 MFLOPs, outperforming GhostNetV1’s 74.5% at similar cost.
• DFC attention provides long-range spatial modeling with lower practical latency than vanilla self-attention on mobile hardware.
• Enhancing the expanded features (expressiveness) via DFC attention yields larger gains (≈1.4% top-1) than enhancing the output features alone.
• GhostNetV2 delivers strong performance across downstream tasks including COCO object detection and ADE20K semantic segmentation, with notable gains over GhostNetV1.
• Practical latency remains favorable on ARM hardware (e.g., ~37 ms with 167 MFLOPs at comparable accuracy).