[Paper Review] Pushing the Limits of Deep CNNs for Pedestrian Detection

Qichang Hu, Peng Wang+3 more|arXiv (Cornell University)|Mar 15, 2016

Advanced Neural Network Applications39 references4 citations

TL;DR

This paper proposes a simple yet highly effective pedestrian detection framework that reuses convolutional feature maps (CFMs) from a fine-tuned VGG-16 network as input to boosted decision forests, achieving state-of-the-art performance on the Caltech pedestrian dataset with a log-average miss rate of 8.93%, a 24% relative improvement over prior work, without requiring complex custom learning architectures or extensive hand-crafted features.

ABSTRACT

Compared to other applications in computer vision, convolutional neural networks have under-performed on pedestrian detection. A breakthrough was made very recently by using sophisticated deep CNN models, with a number of hand-crafted features, or explicit occlusion handling mechanism. In this work, we show that by re-using the convolutional feature maps (CFMs) of a deep convolutional neural network (DCNN) model as image features to train an ensemble of boosted decision models, we are able to achieve the best reported accuracy without using specially designed learning algorithms. We empirically identify and disclose important implementation details. We also show that pixel labelling may be simply combined with a detector to boost the detection performance. By adding complementary hand-crafted features such as optical flow, the DCNN based detector can be further improved. We set a new record on the Caltech pedestrian dataset, lowering the log-average miss rate from $11.7\%$ to $8.9\%$, a relative improvement of $24\%$. We also achieve a comparable result to the state-of-the-art approaches on the KITTI dataset.

Motivation & Objective

• To investigate whether complex deep CNN architectures are necessary for state-of-the-art pedestrian detection performance.
• To evaluate the effectiveness of reusing convolutional feature maps (CFMs) from pre-trained and fine-tuned deep networks as input features for traditional machine learning detectors.
• To explore the impact of multi-layer CFM fusion and integration with pixel-level semantic labeling for performance improvement.
• To demonstrate that simple, ensemble-based models using CFMs can surpass sophisticated DCNN-based detectors without custom learning frameworks.

Proposed method

• Fine-tune a VGG-16 model on the Caltech pedestrian dataset to adapt deep features to pedestrian detection.
• Extract CFMs from specific convolutional layers (e.g., Conv3-3, Conv4-3, Conv5-3) of the fine-tuned VGG-16 as image-level features.
• Train boosted decision forests (e.g., XGBoost or similar) on these CFM features to form a strong baseline detector.
• Combine multiple CFM-based detectors via score averaging to form an ensemble model.
• Integrate semantic pixel labeling scores with detection scores to improve localization and robustness.
• Supplement the CFM-based detector with hand-crafted features such as optical flow to further boost performance.

Experimental results

Research questions

• RQ1 Can a simple, non-custom DCNN feature reuse strategy outperform complex, state-of-the-art DCNN-based pedestrian detectors?
• RQ2 How effective are individual and multi-layer convolutional feature maps (CFMs) from a fine-tuned VGG-16 for pedestrian detection?
• RQ3 To what extent can pixel-level semantic labeling improve the performance of a CFM-based pedestrian detector?
• RQ4 Does combining CFMs with hand-crafted features like optical flow lead to significant performance gains?
• RQ5 Can a lightweight ensemble of boosted trees on CFMs achieve SOTA performance without end-to-end training or complex architectures?

Key findings

• The proposed method achieves a new state-of-the-art log-average miss rate of 8.93% on the Caltech pedestrian dataset, a 24% relative improvement over the prior best of 11.75%.
• Using only the CFM from the Conv3-3 layer of a fine-tuned VGG-16, the model achieves a 13.49% miss rate, outperforming all prior methods except the most complex DCNN frameworks.
• Fusing CFMs from three layers (Conv3-3, Conv4-3, Conv5-3) improves performance to 10.46% miss rate, demonstrating the benefit of multi-scale feature representation.
• The addition of pixel labeling scores to the detection scores further reduces the miss rate to 9.53%, showing the complementary nature of semantic segmentation cues.
• The All-in-one model, combining CFMs, pixel labeling, and hand-crafted features (e.g., optical flow), achieves the best result of 8.93% miss rate, surpassing both CompACT-Deep and DeepParts on Caltech.
• On the KITTI dataset, the fast ensemble model achieves 63.26% AP on the Moderate set, outperforming all monocular-based methods and surpassing CompACT-Deep and DeepParts by over 4.5 percentage points.