[Paper Review] MobileDets: Searching for Object Detection Architectures for Mobile Accelerators

Yunyang Xiong, Hanxiao Liu+8 more|arXiv (Cornell University)|Apr 30, 2020

Advanced Neural Network ApplicationsComputer Science44 references41 citations

TL;DR

MobileDets introduces a platform-aware NAS search space that includes regular convolutions alongside inverted bottlenecks, yielding state-of-the-art latency-accuracy trade-offs for mobile object detection across CPUs, EdgeTPUs, DSPs, and edge GPUs.

ABSTRACT

Inverted bottleneck layers, which are built upon depthwise convolutions, have been the predominant building blocks in state-of-the-art object detection models on mobile devices. In this work, we investigate the optimality of this design pattern over a broad range of mobile accelerators by revisiting the usefulness of regular convolutions. We discover that regular convolutions are a potent component to boost the latency-accuracy trade-off for object detection on accelerators, provided that they are placed strategically in the network via neural architecture search. By incorporating regular convolutions in the search space and directly optimizing the network architectures for object detection, we obtain a family of object detection models, MobileDets, that achieve state-of-the-art results across mobile accelerators. On the COCO object detection task, MobileDets outperform MobileNetV3+SSDLite by 1.7 mAP at comparable mobile CPU inference latencies. MobileDets also outperform MobileNetV2+SSDLite by 1.9 mAP on mobile CPUs, 3.7 mAP on Google EdgeTPU, 3.4 mAP on Qualcomm Hexagon DSP and 2.7 mAP on Nvidia Jetson GPU without increasing latency. Moreover, MobileDets are comparable with the state-of-the-art MnasFPN on mobile CPUs even without using the feature pyramid, and achieve better mAP scores on both EdgeTPUs and DSPs with up to 2x speedup. Code and models are available in the TensorFlow Object Detection API: https://github.com/tensorflow/models/tree/master/research/object_detection.

Motivation & Objective

• Motivate the need to re-evaluate building blocks for mobile detectors beyond inverted bottlenecks (IBNs) on modern accelerators.
• Propose an augmented search space (MobileDet) that includes regular convolutions and Tucker-based blocks to improve latency-accuracy.
• Demonstrate that architecture search directly on object detection tasks yields better results than backbone-only NAS for mobile hardware.
• Show that MobileDets achieve state-of-the-art or competitive mAP at low latency across multiple hardware platforms.
• Provide release-ready code and models within TensorFlow Object Detection API for broader adoption.

Proposed method

• Introduce the MobileDet search space that augments IBN with regular convolutions (Fused Inverted Bottleneck and Tucker/convolution_blocks).
• Describe two flexible building blocks: (i) fused inverted bottleneck (regular KxK conv replacing depthwise+pointwise) and (ii) Tucker convolution (compression via 1x1, KxK, 1x1 blocks).
• Embed these blocks in a latency-aware neural architecture search (NAS) framework (TuNAS) with a platform-aware reward combining mAP and latency.
• Train a cost model c(·) to predict hardware latency from layer decisions, enabling fast NAS without on-device benchmarks for every candidate.
• Search on COCO with detection-specific objectives (SSDLite head) and evaluate final architectures by retraining from scratch on each target hardware.
• Report latency benchmarks using TF-Lite, EdgeTPU, DSP, and GPU backends.

Experimental results

Research questions

• RQ1 Can regular convolutions placed strategically via NAS improve latency-accuracy trade-offs for mobile object detection across diverse hardware?
• RQ2 Does expanding the search space beyond IBN layers (to include fused regular convolutions and Tucker blocks) yield measurable gains on CPUs, EdgeTPUs, DSPs, and GPUs?
• RQ3 Are architectures discovered for one hardware platform transferable to others, and to what extent?
• RQ4 How well does detection-tailored NAS (as opposed to backbone-only NAS) perform on COCO across multiple edge devices?
• RQ5 Can the proposed MobileDet space generalize to unseen hardware (e.g., NVIDIA Jetson GPUs) while maintaining gains?

Key findings

• MobileDets consistently improve latency-accuracy trade-offs across CPUs, EdgeTPUs, DSPs, and edge GPUs compared to baselines relying on IBN-only search spaces.
• On COCO, MobileDets outperform MobileNetV2+SSDLite by 1.7 mAP at comparable CPU latency and outperform MobileNetV2 by 1.9 mAP on mobile CPU, 3.7 mAP on EdgeTPU, 3.4 mAP on DSP, and 2.7 mAP on edge GPU without increasing latency.
• MobileDets are competitive with or exceed the performance of MnasFPN on mobile CPUs, and achieve better mAP on EdgeTPUs and DSPs with up to 2x speedups, even without NAS-FPN heads.
• Including regular convolutions in the search space yields notable gains on non-CPU accelerators (EdgeTPU, DSP) where depthwise convolutions are less optimized.
• Architectures discovered for EdgeTPU/DSP transfer reasonably well to unseen hardware (e.g., NVIDIA Jetson Xavier GPU) demonstrating generality of the MobileDet space.
• The search space with Tucker compression and fused blocks (IBN+Fused+Tucker) provides additional performance gains over IBN-only or smaller spaces on non-CPU hardware.