[Paper Review] A Unified Multi-scale Deep Convolutional Neural Network for Fast Object Detection

Zhaowei Cai, Quanfu Fan+2 more|arXiv (Cornell University)|Jul 25, 2016

Advanced Neural Network Applications35 references4 citations

TL;DR

This paper proposes a unified multi-scale deep CNN (MS-CNN) for fast, accurate object detection by performing detection at multiple intermediate network layers, each optimized for different object scales. By leveraging multi-scale feature maps and deconvolution-based feature upsampling instead of input image upsampling, MS-CNN achieves state-of-the-art performance at up to 15 fps on KITTI and Caltech benchmarks, significantly reducing memory and computation costs while improving detection of small and occluded objects.

ABSTRACT

A unified deep neural network, denoted the multi-scale CNN (MS-CNN), is proposed for fast multi-scale object detection. The MS-CNN consists of a proposal sub-network and a detection sub-network. In the proposal sub-network, detection is performed at multiple output layers, so that receptive fields match objects of different scales. These complementary scale-specific detectors are combined to produce a strong multi-scale object detector. The unified network is learned end-to-end, by optimizing a multi-task loss. Feature upsampling by deconvolution is also explored, as an alternative to input upsampling, to reduce the memory and computation costs. State-of-the-art object detection performance, at up to 15 fps, is reported on datasets, such as KITTI and Caltech, containing a substantial number of small objects.

Motivation & Objective

• To address the challenge of detecting objects at multiple scales efficiently in real-time object detection.
• To overcome the limitations of fixed receptive fields in RPN-based detectors like Faster R-CNN, which underperform on small objects.
• To reduce computational and memory costs associated with input image upsampling used in prior methods for small object detection.
• To improve detection accuracy for small and occluded objects by combining complementary detectors across multiple feature map scales.
• To enable end-to-end training of a unified network that jointly optimizes proposal generation and detection.

Proposed method

• The MS-CNN architecture consists of a shared backbone with a proposal sub-network and a detection sub-network, both trained end-to-end.
• Detection is performed at multiple intermediate layers (e.g., conv-3, conv-5), each with receptive fields matched to specific object scale ranges.
• Feature maps from different layers are used to generate scale-specific object proposals, which are then combined to form a robust multi-scale detector.
• Deconvolutional layers are introduced to upsample feature maps, enhancing responses for small objects without requiring input image upsampling.
• A context encoding module and dimensionality reduction convolution are used to improve feature representation while minimizing parameter growth.
• Hard negative mining and multi-task loss optimization are applied to improve training efficiency and detection accuracy.

Experimental results

Research questions

• RQ1 Can multi-scale detection be effectively performed within a single unified deep network by exploiting intermediate feature maps with varying receptive fields?
• RQ2 Does deconvolution-based feature upsampling outperform input image upsampling in terms of speed, memory efficiency, and detection accuracy for small objects?
• RQ3 Can a unified network architecture achieve state-of-the-art detection performance while maintaining real-time inference speeds of 10–15 fps on datasets with high scale variation?
• RQ4 How does combining detectors from multiple feature map scales improve recall and precision, especially for small and occluded objects?
• RQ5 To what extent does feature map approximation via deconvolution reduce computational cost without degrading detection performance?

Key findings

• MS-CNN achieves a recall of over 95% with only 100 proposals on the KITTI dataset, demonstrating high-quality object proposal generation.
• On the KITTI benchmark, MS-CNN sets new state-of-the-art results for pedestrian and cyclist detection, outperforming Faster R-CNN and 3DOP by 6–7 points on the moderate difficulty set.
• The model achieves 10 fps on KITTI (1250×375) and 15 fps on Caltech (640×480) without input image upsampling, significantly faster than 3DOP (8× faster).
• Deconvolution-based feature upsampling reduces memory and computation costs compared to input upsampling, with minimal impact on accuracy.
• On the Caltech pedestrian benchmark, MS-CNN outperforms DeepParts and other leading methods, especially in medium and partial occlusion scenarios.
• The use of context encoding and dimensionality reduction improves performance while keeping model parameters manageable, with only a small increase in parameter count.