[Paper Review] ImageNet pre-trained models with batch normalization

Marcel Simon, Erik Rodner+1 more|arXiv (Cornell University)|Dec 5, 2016

Advanced Neural Network ApplicationsComputer Science29 references136 citations

TL;DR

The paper provides a new set of ImageNet pre-trained models for Caffe, including batch normalization variants of AlexNet and VGG19 and ResNet models, showing improved performance over previous public models. It also discusses training details and practical guidance for using batch normalization in large CNNs.

ABSTRACT

Convolutional neural networks (CNN) pre-trained on ImageNet are the backbone of most state-of-the-art approaches. In this paper, we present a new set of pre-trained models with popular state-of-the-art architectures for the Caffe framework. The first release includes Residual Networks (ResNets) with generation script as well as the batch-normalization-variants of AlexNet and VGG19. All models outperform previous models with the same architecture. The models and training code are available at http://www.inf-cv.uni-jena.de/Research/CNN+Models.html and https://github.com/cvjena/cnn-models

Motivation & Objective

• Motivate the release of pre-trained ImageNet models for common architectures in Caffe.
• Demonstrate the benefits of adding batch normalization to large CNNs like AlexNet and VGG19.
• Provide training code and practical guidelines to reproduce and fine-tune these models.
• Show empirical gains over prior public models using BN variants and ResNets.

Proposed method

• Modify AlexNet and VGG19 by inserting batch normalization after each convolutional and inner-product layer.
• Remove local response normalization and dropout, and replace mean subtraction with input BN.
• Train networks on ILSVRC 2012 (1.2M images, 1000 classes) for 64 epochs with batch size 256 and distinct initial learning rates per model (0.05 for AlexNet, 0.01 for VGG19, 0.1 for ResNet).
• Use linear learning rate decay and rely on BN to enable larger learning rates and stable statistics; enable global statistics during fine-tuning if GPU memory is limited.

Experimental results

Research questions

• RQ1 Do batch normalization variants of AlexNet and VGG19, as well as ResNets, improve ImageNet pre-training performance over existing public models?
• RQ2 How does batch normalization affect training dynamics and final error rates for large CNNs on ImageNet?
• RQ3 Can fine-tuning BN-enabled models be done without manual mean-subtraction and with robust statistics?
• RQ4 What practical training settings (batch size, learning rate, data augmentation) are required to train BN-enhanced architectures effectively?

Key findings

• BN variants of AlexNet and VGG19 achieve lower top-1 and top-5 error rates than previously published models.
• ResNet models with batch normalization perform competitively with, and in some cases improve upon, prior public residual networks.
• All provided models outperform previously trained models of the same architectures on ILSVRC 2012 validation."
• BN enables using higher learning rates and can improve generalization for large CNNs.
• Training BN-enabled networks without mean subtraction is possible due to input BN layers.