[Paper Review] An Analysis of Pre-Training on Object Detection

Hengduo Li, Bharat Singh+3 more|arXiv (Cornell University)|Apr 11, 2019

Advanced Neural Network Applications59 references33 citations

TL;DR

The paper analyzes how pre-training CNNs for object detection (vs. image classification) affects transfer to detection, segmentation, and classification, showing detection pre-training excels for localization tasks but can hurt classification.

ABSTRACT

We provide a detailed analysis of convolutional neural networks which are pre-trained on the task of object detection. To this end, we train detectors on large datasets like OpenImagesV4, ImageNet Localization and COCO. We analyze how well their features generalize to tasks like image classification, semantic segmentation and object detection on small datasets like PASCAL-VOC, Caltech-256, SUN-397, Flowers-102 etc. Some important conclusions from our analysis are --- 1) Pre-training on large detection datasets is crucial for fine-tuning on small detection datasets, especially when precise localization is needed. For example, we obtain 81.1% mAP on the PASCAL-VOC dataset at 0.7 IoU after pre-training on OpenImagesV4, which is 7.6% better than the recently proposed DeformableConvNetsV2 which uses ImageNet pre-training. 2) Detection pre-training also benefits other localization tasks like semantic segmentation but adversely affects image classification. 3) Features for images (like avg. pooled Conv5) which are similar in the object detection feature space are likely to be similar in the image classification feature space but the converse is not true. 4) Visualization of features reveals that detection neurons have activations over an entire object, while activations for classification networks typically focus on parts. Therefore, detection networks are poor at classification when multiple instances are present in an image or when an instance only covers a small fraction of an image.

Motivation & Objective

• Assess how object-detection pre-training generalizes to detection, segmentation, and classification tasks.
• Compare detection-pre-training to classification-pre-training across multiple target datasets.
• Understand internal feature representations and localization versus classification capabilities.
• Evaluate how dataset size and bounding-box annotations influence transfer performance.

Proposed method

• Pre-train CNN backbones on large detection datasets (OpenImages, ImageNet-Loc, COCO) and compare to ImageNet-Cls pre-training.
• Fine-tune the pre-trained models on Pascal-Voc for detection and segmentation, and on Caltech-256, Sun-397, and Oxford-102 Flowers for classification.
• Use SNIPER-era detectors and standard fine-tuning schedules with data augmentation.
• Analyze localization performance at multiple IoU thresholds (0.1 to 0.7) and quantify boundary versus non-boundary behavior via trimap experiments.
• Visualize activations and perform feature-space analyses (e.g., Conv5 similarities, t-SNE, clustering) to compare detection vs. classification representations.

Experimental results

Research questions

• RQ1 Does pre-training on large detection datasets improve fine-tuning performance for small detection datasets and high IoU localization?
• RQ2 How does detection pre-training affect performance on semantic segmentation and image classification?
• RQ3 What are the internal differences in features and activations between detection-pre-trained and classification-pre-trained networks?
• RQ4 Is similarity in detection feature space predictive of similarity in classification feature space?
• RQ5 How does pre-training dataset size and bounding-box richness influence transfer dynamics?

Key findings

• Pre-training on large detection datasets can improve Pascal-Voc object-detection mAP at high IoU by up to about 7.6% relative to ImageNet pre-training (e.g., 81.1% mAP at 0.7 IoU with OpenImages).
• Detection pre-training yields about a 3% gain on semantic segmentation on Pascal-Voc 2012 compared to classification pre-training.
• Detection pre-training hurts image classification performance by about 8% on Caltech-256 when using detection features for classification without fine-tuning.
• If two images have similar detection features, they are likely to have similar classification features, but not vice versa.
• Activation visualization shows detection networks use whole-object representations while classification networks focus on discriminative parts, impacting robustness to occlusion and multiple instances.