[Paper Review] Fast Object Localization Using a CNN Feature Map Based Multi-Scale Search

Hyungtae Lee, Heesung Kwon+2 more|arXiv (Cornell University)|Apr 12, 2016

Advanced Neural Network Applications18 references3 citations

TL;DR

This paper proposes a fast, real-time object localization method using a multi-scale and multi-aspect ratio search on deep CNN feature maps, bypassing region proposals. By employing dedicated fully connected 'expert' units for different sub-window sizes and aspect ratios, the approach achieves 4 fps inference speed and improves mAP by 2.4% on PASCAL VOC 12 and 2.6% on MSCOCO over prior methods, while significantly reducing computation time.

ABSTRACT

Object localization is an important task in computer vision but requires a large amount of computational power due mainly to an exhaustive multiscale search on the input image. In this paper, we describe a near real-time multiscale search on a deep CNN feature map that does not use region proposals. The proposed approach effectively exploits local semantic information preserved in the feature map of the outermost convolutional layer. A multi-scale search is performed on the feature map by processing all the sub-regions of different sizes using separate expert units of fully connected layers. Each expert unit receives as input local semantic features only from the corresponding sub-regions of a specific geometric shape. Therefore, it contains more nearly optimal parameters tailored to the corresponding shape. This multi-scale and multi-aspect ratio scanning strategy can effectively localize a potential object of an arbitrary size. The proposed approach is fast and able to localize objects of interest with a frame rate of 4 fps while providing improved detection performance over the state-of-the art on the PASCAL VOC 12 and MSCOCO data sets.

Motivation & Objective

• To address the high computational cost of exhaustive multi-scale object localization in deep CNNs.
• To improve localization accuracy without relying on region proposals or slow R-CNN-style pipelines.
• To enable real-time inference (4 fps) for object localization in video or streaming applications.
• To explore the effectiveness of shape-specific expert networks for different object scales and aspect ratios.
• To reduce computation time while maintaining or improving detection performance compared to existing CNN-based localization methods.

Proposed method

• The method performs multi-scale and multi-aspect ratio scanning on the feature map from the last convolutional layer of a pre-trained CNN.
• It uses separate fully connected 'expert' units for each sub-window size and aspect ratio, each trained to classify objects of that specific geometric configuration.
• Sub-regions of interest are extracted from the feature map at multiple scales, with windows ranging from 4×4 to 6×6 in the 6×6 feature map and up to 13×13 in a two-level image pyramid.
• The approach leverages a two-level image pyramid (original and double-sized interpolated image) to cover a broader range of object scales.
• Classification scores from all expert units are aggregated to identify the bounding box with the highest confidence score for each object category.
• A confidence score for each spatial location is computed as the average of normalized scores from all detections containing that location, using n=5 to suppress low-confidence detections.

Experimental results

Research questions

• RQ1 Can a multi-scale and multi-aspect ratio scanning strategy on CNN feature maps improve object localization accuracy without using region proposals?
• RQ2 Does using dedicated expert fully connected layers for different window shapes lead to better localization performance than a single shared classifier?
• RQ3 Can this approach achieve real-time inference (≥4 fps) while outperforming state-of-the-art methods on standard benchmarks?
• RQ4 How does the use of a two-level image pyramid affect the coverage of object scales and localization performance?
• RQ5 What is the trade-off between accuracy and inference speed when increasing the number of pyramid levels?

Key findings

• The proposed method achieves a frame rate of 4 fps, significantly faster than RCNN (9.0 sec/im) and Fast-RCNN (2.1 sec/im), and even faster than Oquab15 (1.3 sec/im).
• On the PASCAL VOC 2012 validation set, the method achieves 75.4% mAP, outperforming RCNN (74.8%) and Fast-RCNN (71.3%).
• With a three-level image pyramid, the method improves mAP by 2.6% for classification and 2.4% for localization over the two-level version, though at the cost of increased inference time (1.58 sec/im).
• Using multiple expert units improves localization mAP by 5.3 percentage points over a single shared fully connected layer (77.8% vs. 72.5%).
• The method outperforms the combination of Oquab15 with selective search proposals in localization accuracy, demonstrating the effectiveness of the sub-window-based approach.
• The confidence score aggregation method using n=5 normalization effectively suppresses low-confidence detections and improves localization robustness.