[Paper Review] Rethinking ImageNet Pre-training

Kaiming He, Ross Girshick+1 more|arXiv (Cornell University)|Nov 21, 2018

Advanced Neural Network Applications39 references99 citations

TL;DR

The paper demonstrates that object detection and instance segmentation on COCO can match or exceed ImageNet-pretrained models when trained from scratch, given normalization, longer training, and appropriate hyper-parameter tuning; ImageNet pre-training mainly speeds up early convergence and is not always necessary for final accuracy.

ABSTRACT

We report competitive results on object detection and instance segmentation on the COCO dataset using standard models trained from random initialization. The results are no worse than their ImageNet pre-training counterparts even when using the hyper-parameters of the baseline system (Mask R-CNN) that were optimized for fine-tuning pre-trained models, with the sole exception of increasing the number of training iterations so the randomly initialized models may converge. Training from random initialization is surprisingly robust; our results hold even when: (i) using only 10% of the training data, (ii) for deeper and wider models, and (iii) for multiple tasks and metrics. Experiments show that ImageNet pre-training speeds up convergence early in training, but does not necessarily provide regularization or improve final target task accuracy. To push the envelope we demonstrate 50.9 AP on COCO object detection without using any external data---a result on par with the top COCO 2017 competition results that used ImageNet pre-training. These observations challenge the conventional wisdom of ImageNet pre-training for dependent tasks and we expect these discoveries will encourage people to rethink the current de facto paradigm of `pre-training and fine-tuning' in computer vision.

Motivation & Objective

• Question the necessity of ImageNet pre-training for object detection and segmentation on COCO.
• Evaluate whether training from scratch can achieve comparable or superior final performance under standard baselines and hyperparameters.
• Identify normalization and training-length adjustments required to enable scratch training across architectures and data regimes.
• Assess how data scale (full COCO vs. reduced data) affects the relative benefits of pre-training.

Proposed method

• Use Mask R-CNN with ResNet/ResNeXt backbones and FPN on COCO train2017, evaluating bbox and mask AP on val2017.
• Replace frozen BatchNorm with GroupNorm or SyncBN to enable stable scratch training.
• Increase training iterations (6× schedule) to allow scratch models to converge.
• Use training-time scale augmentation and data augmentation to study robustness across data regimes.
• Compare scratch vs. ImageNet pre-training across various architectures, data scales, and task-specific metrics (bbox AP, mask AP, keypoint AP).
• Demonstrate large-scale scratch training (X152 with GN) achieving high AP without pre-training.

Experimental results

Research questions

• RQ1 Can object detection and instance segmentation on COCO reach parity with ImageNet-pretrained models when trained from scratch?
• RQ2 What normalization techniques are necessary to enable stable scratch training for detectors?
• RQ3 How does training duration affect scratch convergence and final accuracy compared to pre-training?
• RQ4 Does ImageNet pre-training provide regularization benefits or primarily accelerate early convergence, especially under limited data?
• RQ5 How do scratch-trained models perform on localization-sensitive metrics and keypoint detection?

Key findings

• Scratch training on COCO can match or exceed the accuracy of ImageNet-pretrained models for multiple baselines when using GN/SyncBN and extended training schedules.
• ImageNet pre-training speeds up early convergence but does not necessarily improve final target task accuracy under standard schedules; with longer training (5×–6×), scratch models reach comparable or better AP.
• Scratch training remains competitive even with as little as 10% of COCO data, and larger backbones (e.g., X152) trained from scratch can reach ~50.9 bbox AP and ~43.2 mask AP on val2017.
• Localization-sensitive metrics and keypoint detection often show less or no benefit from ImageNet pre-training; scratch models can perform as well or better in high-overlap thresholds and keypoint tasks.
• Across data regimes, proper normalization and longer optimization are crucial; pre-training helps less when data is plentiful or when the task emphasizes localization rather than classification.