[Paper Review] Identity Mappings in Deep Residual Networks

Kaiming He, Xiangyu Zhang+2 more|arXiv (Cornell University)|Mar 16, 2016

Advanced Neural Network Applications22 references210 citations

TL;DR

This paper analyzes how identity skip connections and identity after-addition activation enable direct forward and backward information propagation in very deep ResNets, proposes a pre-activation residual unit, and demonstrates improved training and generalization for extremely deep networks on CIFAR and ImageNet.

ABSTRACT

Deep residual networks have emerged as a family of extremely deep architectures showing compelling accuracy and nice convergence behaviors. In this paper, we analyze the propagation formulations behind the residual building blocks, which suggest that the forward and backward signals can be directly propagated from one block to any other block, when using identity mappings as the skip connections and after-addition activation. A series of ablation experiments support the importance of these identity mappings. This motivates us to propose a new residual unit, which makes training easier and improves generalization. We report improved results using a 1001-layer ResNet on CIFAR-10 (4.62% error) and CIFAR-100, and a 200-layer ResNet on ImageNet. Code is available at: https://github.com/KaimingHe/resnet-1k-layers

Motivation & Objective

• Motivate and analyze how skip connections facilitate information propagation in very deep residual networks.
• Investigate the impact of different shortcut types and activation placements on optimization and generalization.
• Propose a new residual unit with identity after addition (pre-activation) that eases optimization and improves performance.
• Demonstrate state-of-the-art or competitive results with ultra-deep networks on CIFAR-10/100 and ImageNet.
• Provide practical guidelines for designing deep ResNets to balance optimization ease and model capacity.

Proposed method

• Derive forward and backward propagation properties under two identity conditions: identity skip connection h(x)=x and identity after-addition activation f(y)=y.
• Analyze the effect of non-identity shortcuts (scaling, gating, 1x1 conv, dropout) using theoretical expressions and ablation experiments.
• Introduce a pre-activation residual unit where activation functions are moved before weight layers, effectively making the after-addition activation an identity.
• Experimentally compare variants on CIFAR-10/100 with ResNet-110/164/1001 architectures and on ImageNet with ResNet-152/200 variants.
• Provide training and architectural guidelines, including the impact of BN/ReLU placement relative to addition (pre-activation vs post-activation).
• Report performance metrics and training dynamics to support the proposed design choices.

Experimental results

Research questions

• RQ1 How do identity skip connections and identity after-addition activation influence forward signal propagation in deep ResNets?
• RQ2 What is the impact of non-identity shortcut components (scaling, gating, 1x1 convolutions, dropout) on optimization and generalization?
• RQ3 Can a pre-activation residual unit enable training of ultra-deep networks and improve generalization compared to the original residual unit?
• RQ4 How do activation placement and BN timing (pre- vs post-activation) affect performance on CIFAR-10/100 and ImageNet?
• RQ5 What practical guidelines emerge for constructing very deep ResNets that are easier to train and offer better accuracy?

Key findings

• Identity skip connections and identity after-addition activation substantially ease optimization by allowing direct signal propagation across layers.
• Non-identity shortcut components generally impede information flow and worsen training dynamics or final performance.
• A pre-activation residual unit (BN and ReLU applied before weight layers) enables training of extremely deep networks (e.g., 1001 layers) with improved generalization on CIFAR-10/100 and competitive results on ImageNet.
• On CIFAR-10, 1001-layer ResNet achieved 4.62% test error (best result with pre-activation variant); on CIFAR-10/100, pre-activation models consistently outperformed their baseline counterparts (e.g., ResNet-1001 CIFAR-10: 4.92% baseline vs 4.89% ±0.14, CIFAR-100: 22.71% vs 22.68% ±0.22).
• On ImageNet, pre-activation ResNets showed improvements over the original for comparable architectures: ResNet-152 top-1 21.1% vs 21.3% with standard design, and ResNet-200 with pre-activation achieved 20.7% top-1 (320x320 testing) versus 21.8% for the original. When augmented, pre-activation ResNet-200 reached 20.1% top-1 (with scale+aspect augmentation) and 4.8% top-5.