[Paper Review] Bag of Tricks for Image Classification with Convolutional Neural Networks

Tong He, Zhang Zhi+4 more|arXiv (Cornell University)|Dec 4, 2018

Advanced Neural Network ApplicationsComputer Science26 references142 citations

TL;DR

The paper surveys and empirically evaluates a collection of training procedure refinements and minor architecture tweaks, showing that stacking them improves ImageNet accuracy and transfer learning performance.

ABSTRACT

Much of the recent progress made in image classification research can be credited to training procedure refinements, such as changes in data augmentations and optimization methods. In the literature, however, most refinements are either briefly mentioned as implementation details or only visible in source code. In this paper, we will examine a collection of such refinements and empirically evaluate their impact on the final model accuracy through ablation study. We will show that, by combining these refinements together, we are able to improve various CNN models significantly. For example, we raise ResNet-50's top-1 validation accuracy from 75.3% to 79.29% on ImageNet. We will also demonstrate that improvement on image classification accuracy leads to better transfer learning performance in other application domains such as object detection and semantic segmentation.

Motivation & Objective

• Identify a set of training procedure refinements and minor architecture tweaks that improve CNN accuracy without increasing computational cost.
• Quantitatively assess the impact of each trick through ablation studies on multiple networks and datasets.
• Demonstrate that combined tricks outperform newer architectures under standard pipelines.
• Show that accuracy gains translate to better transfer learning performance in object detection and semantic segmentation.

Proposed method

• Define a baseline ResNet training procedure and neighborhood of tricks for data preprocessing, optimization, and regularization.
• Conduct ablation studies to isolate the impact of each trick on Top-1/Top-5 accuracy.
• Evaluate tricks on ResNet-50, Inception-V3, and MobileNet on ImageNet; report FLOPs and accuracy.
• Propose several model tweaks for ResNet architectures to assess their effect on accuracy and FLOPs.
• Investigate large-batch training, mixed-precision training, learning rate schedules, label smoothing, distillation, and mixup.
• Test transfer learning performance for object detection (Faster-RCNN on PASCAL VOC) and semantic segmentation (ADE20K).

Experimental results

Research questions

• RQ1 What is the impact of individual training refinements on CNN accuracy on ImageNet?
• RQ2 How does stacking multiple tricks compare to standard training pipelines across architectures?
• RQ3 Do accuracy gains from tricks transfer to other tasks like object detection and semantic segmentation?
• RQ4 Which combination of tricks yields the best trade-off between accuracy and computational cost?

Key findings

• ResNet-50 with all tricks achieves 79.29% top-1 on ImageNet, surpassing several newer architectures under the standard pipeline.
• Combining tricks yields larger gains than applying them individually in most cases.
• Tricks generalize to Inception-V3 and MobileNet and improve transfer learning performance in object detection and semantic segmentation.
• Large-batch and mixed-precision training significantly reduce training time and can maintain or improve accuracy when combined with the tricks.
• Knowledge distillation is beneficial for some architectures (notably ResNet families) but not universally across all models.
• Label smoothing, mixup, and cosine learning rate decay contribute notable improvements in validation accuracy on ImageNet.