[Paper Review] HyperNet: Towards Accurate Region Proposal Generation and Joint Object Detection

Tao Kong, Anbang Yao+2 more|arXiv (Cornell University)|Apr 3, 2016

Advanced Neural Network Applications139 citations

TL;DR

HyperNet jointly learns region proposals and object detection by fusing hierarchical CNN features into a Hyper Feature, achieving high recall with about 100 proposals and state-of-the-art mAP on VOC2007/2012, with real-time potential.

ABSTRACT

Almost all of the current top-performing object detection networks employ region proposals to guide the search for object instances. State-of-the-art region proposal methods usually need several thousand proposals to get high recall, thus hurting the detection efficiency. Although the latest Region Proposal Network method gets promising detection accuracy with several hundred proposals, it still struggles in small-size object detection and precise localization (e.g., large IoU thresholds), mainly due to the coarseness of its feature maps. In this paper, we present a deep hierarchical network, namely HyperNet, for handling region proposal generation and object detection jointly. Our HyperNet is primarily based on an elaborately designed Hyper Feature which aggregates hierarchical feature maps first and then compresses them into a uniform space. The Hyper Features well incorporate deep but highly semantic, intermediate but really complementary, and shallow but naturally high-resolution features of the image, thus enabling us to construct HyperNet by sharing them both in generating proposals and detecting objects via an end-to-end joint training strategy. For the deep VGG16 model, our method achieves completely leading recall and state-of-the-art object detection accuracy on PASCAL VOC 2007 and 2012 using only 100 proposals per image. It runs with a speed of 5 fps (including all steps) on a GPU, thus having the potential for real-time processing.

Motivation & Objective

• Motivate reducing the number of region proposals while maintaining high recall.
• Develop a unified network that jointly optimizes region proposal generation and object detection.
• Leverage multi-level CNN features to improve small object localization and localization precision.
• Propose an efficient training and inference framework suitable for real-time or large-scale deployment.

Proposed method

• Create Hyper Feature by aggregating and compressing hierarchical CNN feature maps from multiple layers into a uniform space.
• Design a lightweight region proposal network that outputs about 100 proposals per image with ROI pooling and bounding box regression.
• Implement a detection network that shares Hyper Feature, uses an enhanced Conv layer before FC, and performs class-specific bounding box regression with NMS.
• Train proposal and detection modules jointly through a 6-step training procedure to form a unified HyperNet.
• Speed up the proposal and detection stages by reordering layers to reduce feature dimensionality and simplify the classifier.

Experimental results

Research questions

• RQ1 Can HyperNet achieve high recall with a small number of proposals (around 100) across IoU thresholds?
• RQ2 Does fusing deep, intermediate, and shallow CNN features (Hyper Feature) improve both proposal quality and detection accuracy, especially for small objects?
• RQ3 How does joint training of proposal generation and object detection affect overall performance compared to staged training?
• RQ4 What are the runtime characteristics of HyperNet, and can it approach real-time performance without sacrificing accuracy?

Key findings

• HyperNet achieves 95% recall with 50 proposals and 97% recall with 100 proposals at IoU 0.5 on PASCAL VOC 2007.
• On VOC 2007, HyperNet attains 76.3% mAP, surpassing Fast R-CNN by 6.3 points and Faster R-CNN by 3.1 points under IoU 0.5.
• On VOC 2012, HyperNet achieves 71.4% mAP (top result in the comp4 track), outperforming several baselines.
• The speeding-up version (HyperNet-SP) maintains strong accuracy while achieving approximately 5 fps on GPU hardware.
• The architecture’s Hyper Feature, which combines multi-level features with appropriate resolution, contributes to improved localization and small-object detection (e.g., bottles, plants).