[Paper Review] ThunderNet: Towards Real-time Generic Object Detection

Zheng Qin, Zeming Li+5 more|arXiv (Cornell University)|Mar 28, 2019

Advanced Neural Network ApplicationsComputer Science31 references43 citations

TL;DR

ThunderNet presents a lightweight two-stage detector designed for real-time generic object detection on mobile devices, with a custom lightweight backbone (SNet) and efficient detection head including Context Enhancement Module and Spatial Attention Module, achieving ARM real-time speeds with competitive accuracy at low FLOPs.

ABSTRACT

Real-time generic object detection on mobile platforms is a crucial but challenging computer vision task. However, previous CNN-based detectors suffer from enormous computational cost, which hinders them from real-time inference in computation-constrained scenarios. In this paper, we investigate the effectiveness of two-stage detectors in real-time generic detection and propose a lightweight two-stage detector named ThunderNet. In the backbone part, we analyze the drawbacks in previous lightweight backbones and present a lightweight backbone designed for object detection. In the detection part, we exploit an extremely efficient RPN and detection head design. To generate more discriminative feature representation, we design two efficient architecture blocks, Context Enhancement Module and Spatial Attention Module. At last, we investigate the balance between the input resolution, the backbone, and the detection head. Compared with lightweight one-stage detectors, ThunderNet achieves superior performance with only 40% of the computational cost on PASCAL VOC and COCO benchmarks. Without bells and whistles, our model runs at 24.1 fps on an ARM-based device. To the best of our knowledge, this is the first real-time detector reported on ARM platforms. Our code and models are available at \url{https://github.com/qinzheng93/ThunderNet}.

Motivation & Objective

• Investigate whether two-stage detectors can achieve real-time performance on mobile devices.
• Design a lightweight backbone specialized for object detection rather than transfer from image classification.
• Develop efficient detection head components to balance accuracy and computational cost.
• Bridge input resolution, backbone capacity, and detection head design for optimal real-time performance.

Proposed method

• Propose SNet lightweight backbones by modifying ShuffleNetV2 with 5x5 depthwise convolutions to enlarge receptive fields.
• Compress RPN and RoI head components to reduce computation while preserving accuracy (e.g., 5x5 depthwise in RPN, 1x1 conv, and reduced R-CNN fc size).
• Introduce Context Enhancement Module (CEM) to fuse multi-scale local and global context (C4, C5, Cglb) with 1x1 projections and upsampling/broadcasting.
• Introduce Spatial Attention Module (SAM) to re-weight CEM features using RPN-derived foreground signals via a 1x1 transform.
• Explore balance between input resolution, backbone, and detection head to maximize speed and accuracy on mobile hardware.
• Train end-to-end with synchronized SGD, multi-scale training, Cross-GPU Batch Normalization, and Soft-NMS.

Experimental results

Research questions

• RQ1 Can a two-stage detector outperform lightweight one-stage detectors on mobile hardware in terms of speed and accuracy?
• RQ2 What backbone and detection-head design choices yield the best accuracy-efficiency trade-off for real-time mobile detection?
• RQ3 How do context and spatial attention mechanisms impact feature representations and localization?
• RQ4 What is the optimal balance among input resolution, backbone capacity, and detection head complexity for ARM platforms?

Key findings

• ThunderNet with SNet49 achieves MobileNet-SSD level accuracy at ~22% of FLOPs.
• ThunderNet with SNet146 surpasses prior lightweight detectors at ~40% of FLOPs.
• ThunderNet with SNet535 rivals large detectors at a small fraction of FLOPs.
• On COCO test-dev, ThunderNet with SNet146 achieves AP 23.7, AP50 40.3, AP75 24.6; with SNet535 it reaches AP 28.1, AP50 46.2, AP75 29.6.
• ThunderNet runs at 24.1 fps on ARM (SNet49) and 13.8 fps on ARM (SNet146), while over 200 fps on GPU for all variants.
• A large-backbone-small-head design outperforms small-backbone-large-head under similar FLOPs, highlighting backbone-head compatibility.