[Paper Review] Deep Residual Learning for Image Recognition

Kaiming He, Xiangyu Zhang+2 more|arXiv (Cornell University)|Dec 10, 2015

Advanced Neural Network Applications50 references4,626 citations

TL;DR

The paper introduces residual learning with identity shortcut connections to train much deeper networks, enabling up to 152 layers on ImageNet and achieving state-of-the-art results, while addressing degradation in very deep plain nets.

ABSTRACT

Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers---8x deeper than VGG nets but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.

Motivation & Objective

• Motivate the need for deeper networks in visual recognition and identify degradation problems when depth increases.
• Propose a residual learning framework that reformulates layers to learn residual functions with reference to inputs.
• Demonstrate that residual nets are easier to optimize and benefit from increased depth across datasets (ImageNet and CIFAR-10).
• Show that extremely deep residual nets achieve superior accuracy and generalize to detection/localization tasks (COCO, Pascal VOC).
• Provide practical architectures and training strategies that enable successful training of very deep networks.

Proposed method

• Formulate the target mapping H(x) as learning a residual F(x) = H(x) - x, so that the block computes y = F(x) + x.
• Introduce identity shortcut connections that perform simple addition to propagate information without adding parameters or computational cost.
• Explore network variants including plain nets and residual nets, studying depth from 18 to 152 layers and using both non-bottleneck and bottleneck residual blocks.
• Use 3x3 convolutional cores in plain nets and in residual blocks, with shortcut connections either identity or with projection when dimensions change.
• Adopt a bottleneck design (1x1, 3x3, 1x1) for deeper models to keep computational cost reasonable while increasing depth.
• Train with SGD, batch normalization, and standard image augmentation on ImageNet, CIFAR-10, and COCO/Pascal VOC benchmarks; evaluate top-1 and top-5 errors (and mAP in detection).

Experimental results

Research questions

• RQ1 Does the degradation problem in very deep plain networks impede optimization, and can residual learning alleviate it?
• RQ2 Can substantially deeper residual networks (up to 152 layers) improve accuracy on ImageNet and CIFAR-10 compared to shallower equivalents?
• RQ3 How do identity shortcuts compare with projection shortcuts in terms of training ease and performance?
• RQ4 Do extremely deep residual nets generalize to object detection/segmentation tasks (COCO, Pascal VOC) beyond image classification?
• RQ5 What are practical architectural variants (plain vs. residual, bottleneck vs. non-bottleneck) that balance depth, compute, and accuracy?

Key findings

• Deep plain nets exhibit degradation: deeper networks can have higher training error and poorer validation performance.
• Residual nets with skip connections address degradation and achieve higher accuracy as depth increases (e.g., ResNet-34 vs. ResNet-18).
• On ImageNet, ResNet-50/101/152 achieve top-1 errors of 22.85%, 21.75%, and 21.43% respectively (single-model results), and top-5 errors of 6.71%, 6.05%, and 5.71% respectively.
• An ensemble of residual nets achieves 3.57% top-5 error on ImageNet test set, winning first place in ILSVRC 2015 classification.
• Residual nets provide strong gains on CIFAR-10, with ResNet-110 achieving 6.43% (best reported in a single run) and deeper variants achieving further improvements.
• On COCO, replacing VGG-16 with ResNet-101 yields notable gains in detection (mAP improvements), demonstrating strong generalization to other vision tasks.