[Paper Review] DeepID-Net: multi-stage and deformable deep convolutional neural networks for object detection

Wanli Ouyang, Ping Luo+13 more|arXiv (Cornell University)|Sep 11, 2014

Advanced Neural Network Applications75 references134 citations

TL;DR

This paper proposes DeepID-Net, a multi-stage and deformable deep convolutional neural network for generic object detection that improves feature learning, part deformation modeling, and context integration. By introducing a deformation-constrained pooling (def-pooling) layer, a novel pretraining strategy using object-level annotations, multi-stage training, and diverse model averaging, the method achieves a 45% mean average precision on ILSVRC 2014, significantly outperforming RCNN's 31%.

ABSTRACT

In this paper, we propose multi-stage and deformable deep convolutional neural networks for object detection. This new deep learning object detection diagram has innovations in multiple aspects. In the proposed new deep architecture, a new deformation constrained pooling (def-pooling) layer models the deformation of object parts with geometric constraint and penalty. With the proposed multi-stage training strategy, multiple classifiers are jointly optimized to process samples at different difficulty levels. A new pre-training strategy is proposed to learn feature representations more suitable for the object detection task and with good generalization capability. By changing the net structures, training strategies, adding and removing some key components in the detection pipeline, a set of models with large diversity are obtained, which significantly improves the effectiveness of modeling averaging. The proposed approach ranked \#2 in ILSVRC 2014. It improves the mean averaged precision obtained by RCNN, which is the state-of-the-art of object detection, from $31\%$ to $45\%$. Detailed component-wise analysis is also provided through extensive experimental evaluation.

Motivation & Objective

• To improve generic object detection in complex scenarios with large intra-class variation, deformation, and cluttered backgrounds.
• To develop a deep learning framework that jointly optimizes feature representation, part deformation modeling, and contextual reasoning.
• To overcome overfitting and improve generalization in deep object detectors through novel training and pretraining strategies.
• To enhance model performance via effective model averaging using diverse architectures and training schemes.

Proposed method

• Introduces a deformation-constrained pooling (def-pooling) layer that learns geometric deformation of object parts with penalty, replacing standard max-pooling.
• Proposes a new pretraining strategy using object-level annotations from ImageNet 1000-class classification instead of image-level labels, improving feature transfer to detection.
• Employs a multi-stage training scheme where classifiers at each stage handle samples of increasing difficulty, with joint optimization and regularization to reduce overfitting.
• Uses a diverse set of models generated by varying network structures and training strategies for model averaging, enhancing performance through diversity.
• Incorporates contextual information from image classification scores and applies bounding box regression to refine localization.
• Applies model averaging across multiple diverse models, with class-specific combination strategies to further boost detection accuracy.

Experimental results

Research questions

• RQ1 Can a deep learning framework that jointly models feature representation, part deformation, and context improve generic object detection beyond RCNN?
• RQ2 How does pretraining on object-level annotations instead of image-level annotations affect detection performance?
• RQ3 To what extent does a multi-stage training strategy with progressive hard sample mining improve generalization and reduce overfitting?
• RQ4 How effective is model averaging when models are diverse in architecture and training strategy, especially across different object categories?
• RQ5 What is the contribution of each component—def-pooling, context modeling, bounding box regression, and pretraining—to the final detection accuracy?

Key findings

• The proposed DeepID-Net achieved a mean average precision (mAP) of 45% on the ILSVRC 2014 validation set, up from RCNN’s 31%, ranking #2 in the challenge.
• Replacing image-level pretraining with object-level pretraining improved mAP by approximately 4%.
• The def-pooling layer contributed a 2.5% increase in mAP by modeling part deformations with geometric constraints.
• Bounding box regression and contextual information from classification scores each improved mAP by about 1%.
• Model averaging across diverse models (varying architecture and training strategies) significantly enhanced performance, with the final mAP reaching 45%.
• The multi-stage training scheme effectively handled samples of varying difficulty and reduced overfitting compared to standard backpropagation.