[Paper Review] SkyNet: a Hardware-Efficient Method for Object Detection and Tracking on Embedded Systems

Xiaofan Zhang, Haoming Lu+10 more|arXiv (Cornell University)|Sep 20, 2019

Video Surveillance and Tracking MethodsComputer Science67 references67 citations

TL;DR

SkyNet introduces a bottom-up, hardware-aware DNN design for embedded object detection and tracking, achieving state-of-the-art efficiency and accuracy on TX2 GPU and Ultra96 FPGA, and boosting trackers like SiamRPN++ and SiamMask.

ABSTRACT

Object detection and tracking are challenging tasks for resource-constrained embedded systems. While these tasks are among the most compute-intensive tasks from the artificial intelligence domain, they are only allowed to use limited computation and memory resources on embedded devices. In the meanwhile, such resource-constrained implementations are often required to satisfy additional demanding requirements such as real-time response, high-throughput performance, and reliable inference accuracy. To overcome these challenges, we propose SkyNet, a hardware-efficient neural network to deliver the state-of-the-art detection accuracy and speed for embedded systems. Instead of following the common top-down flow for compact DNN (Deep Neural Network) design, SkyNet provides a bottom-up DNN design approach with comprehensive understanding of the hardware constraints at the very beginning to deliver hardware-efficient DNNs. The effectiveness of SkyNet is demonstrated by winning the competitive System Design Contest for low power object detection in the 56th IEEE/ACM Design Automation Conference (DAC-SDC), where our SkyNet significantly outperforms all other 100+ competitors: it delivers 0.731 Intersection over Union (IoU) and 67.33 frames per second (FPS) on a TX2 embedded GPU; and 0.716 IoU and 25.05 FPS on an Ultra96 embedded FPGA. The evaluation of SkyNet is also extended to GOT-10K, a recent large-scale high-diversity benchmark for generic object tracking in the wild. For state-of-the-art object trackers SiamRPN++ and SiamMask, where ResNet-50 is employed as the backbone, implementations using our SkyNet as the backbone DNN are 1.60X and 1.73X faster with better or similar accuracy when running on a 1080Ti GPU, and 37.20X smaller in terms of parameter size for significantly better memory and storage footprint.

Motivation & Objective

• Address the challenge of delivering real-time, high-accuracy object detection and tracking on resource-constrained embedded devices.
• Propose a bottom-up, hardware-aware DNN design that anticipates hardware constraints from the outset.
• Demonstrate SkyNet on embedded GPU and FPGA platforms and show competitive accuracy and energy efficiency.
• Extend SkyNet as a backbone to improve speed and memory footprint of state-of-the-art trackers.

Proposed method

• Define hardware-aware Bundles as basic building blocks of DNNs and evaluate them on target hardware to capture realistic latency and resource usage.
• Stage 1: Bundle selection and evaluation to identify promising components that balance accuracy and hardware cost.
• Stage 2: Hardware-aware DNN search using a group-based particle swarm optimization (PSO) to optimize network configuration under accuracy and latency targets.
• Stage 3: Add advanced features such as feature map bypass, reordering, and ReLU6 to improve small-object detection and hardware efficiency.
• Present SkyNet architecture built from repeated DW-Conv3 and PW-Conv1 blocks with BN and ReLU6, and adapt YOLO-style bounding box regression heads.
• Quantization and tiling/batching strategies for FPGA to balance accuracy, memory, and bandwidth.

Experimental results

Research questions

• RQ1 Can a bottom-up, hardware-aware design approach outperform traditional top-down DNN design flows for embedded detectors?
• RQ2 How can hardware feedback be integrated into the DNN search process to optimize for latency and resource usage on both embedded GPUs and FPGAs?
• RQ3 What architectural features (like feature map bypass and ReLU6) yield the best trade-offs between small-object detection and hardware efficiency?
• RQ4 How does SkyNet perform as a backbone in object tracking pipelines such as SiamRPN++ and SiamMask compared to ResNet-50?
• RQ5 What are the gains in speed and memory footprint when replacing backbone backbones with SkyNet in GOT-10K-style benchmarks?

Key findings

• SkyNet achieved 0.731 IoU and 67.33 FPS on TX2 embedded GPU and 0.716 IoU and 25.05 FPS on Ultra96 FPGA in DAC-SDC, outperforming 100+ competitors.
• SkyNet backbones yield 1.60x and 1.73x speedups for SiamRPN++ and SiamMask on a 1080Ti GPU respectively, with smaller parameter counts (37.20x smaller).
• Ablation shows SkyNet C with ReLU6 (IoU 0.741) as the best validation model on DAC-SDC, and SkyNet backbones reach IoU around 0.73–0.74 with modest parameter footprints (≈0.44M for the base backbone).
• The bottom-up, hardware-aware PSO-based search can identify DNN candidates that meet software accuracy and hardware latency targets, demonstrating efficient hardware-aware exploration.
• FPGA-specific results show quantization and tiling/batching strategies enabling feasible BRAM usage while maintaining accuracy losses within acceptable ranges.