[Paper Review] PVANET: Deep but Lightweight Neural Networks for Real-time Object Detection

Kye-Hyeon Kim, Sang‐Hoon Hong+3 more|arXiv (Cornell University)|Aug 29, 2016

Advanced Neural Network ApplicationsComputer Science7 references202 citations

TL;DR

PVANET designs a deep, thin feature extractor with C.ReLU, Inception, and HyperNet-inspired multi-scale fusion to achieve state-of-the-art mAP on VOC while dramatically reducing computation, enabling real-time detection.

ABSTRACT

This paper presents how we can achieve the state-of-the-art accuracy in multi-category object detection task while minimizing the computational cost by adapting and combining recent technical innovations. Following the common pipeline of "CNN feature extraction + region proposal + RoI classification", we mainly redesign the feature extraction part, since region proposal part is not computationally expensive and classification part can be efficiently compressed with common techniques like truncated SVD. Our design principle is "less channels with more layers" and adoption of some building blocks including concatenated ReLU, Inception, and HyperNet. The designed network is deep and thin and trained with the help of batch normalization, residual connections, and learning rate scheduling based on plateau detection. We obtained solid results on well-known object detection benchmarks: 83.8% mAP (mean average precision) on VOC2007 and 82.5% mAP on VOC2012 (2nd place), while taking only 750ms/image on Intel i7-6700K CPU with a single core and 46ms/image on NVIDIA Titan X GPU. Theoretically, our network requires only 12.3% of the computational cost compared to ResNet-101, the winner on VOC2012.

Motivation & Objective

• Motivate reducing computational cost in object detection without sacrificing accuracy.
• Design a lightweight yet deep feature extractor suitable for Faster R-CNN frameworks.
• Explore building blocks (C.ReLU, Inception, HyperNet) to balance speed and performance.
• Demonstrate real-time performance with competitive mAP on VOC benchmarks.

Proposed method

• Redesign the feature extraction network while keeping region proposal and RoI classification efficient.
• Use C.ReLU to halve early-stage channels and concatenate with negated activations to double feature maps.
• Employ Inception modules to capture multi-scale receptive fields.
• Adopt HyperNet-style multi-scale intermediate outputs fused into a final feature map for RPN and classifier.
• Train with batch normalization, residual connections, and plateau-based learning rate scheduling.
• Feed only a subset of multi-scale features to RPN for efficiency while using full convf for RoI-based classification.

Experimental results

Research questions

• RQ1 Can a deep but narrow network achieve state-of-the-art detection accuracy with substantially lower computational cost?
• RQ2 Do C.ReLU, Inception blocks, and multi-scale fusion (HyperNet-style) provide measurable gains for object detection benchmarks?
• RQ3 What are the real-time performance and accuracy trade-offs on CPU and GPU for PVANET in Faster R-CNN setups?
• RQ4 How does PVANET’s performance compare to ResNet-101 and other backbones in VOC2007/2012 benchmarks?

Key findings

• PVANET achieves 83.8% mAP on VOC2007 and 82.5% mAP on VOC2012 (2nd place) with substantially lower computation than ResNet-101.
• PVANET uses 7.9 GMAC for feature extraction on 1065x640 input, about 12.3% of ResNet-101’s cost.
• On CPU (single core i7-6700K) PVANET runs at 750 ms/image (1.3 FPS); on Titan X GPU it runs at 46 ms/image (21.7 FPS).
• RPN uses only the first 128 channels of convf for proposals, while RCNN uses 512-channel convf for RoI processing.
• PVANET+ (with bounding-box voting) attains 83.8% mAP in VOC2007 and 82.5% mAP in VOC2012, while being more efficient than several heavier backbones.
• Compressing the fully connected layers via truncated SVD reduces runtime with a modest drop in mAP (82.9%).