[Paper Review] Wider or Deeper: Revisiting the ResNet Model for Visual Recognition

Zifeng Wu, Chunhua Shen+1 more|arXiv (Cornell University)|Nov 30, 2016

Advanced Neural Network Applications34 references230 citations

TL;DR

The paper reinterprets ResNets as a linearly growing ensemble of shallower sub-networks, proposes a shallower, wider residual architecture that outperforms much deeper models on ImageNet, and demonstrates strong transfer to semantic segmentation tasks.

ABSTRACT

The trend towards increasingly deep neural networks has been driven by a general observation that increasing depth increases the performance of a network. Recently, however, evidence has been amassing that simply increasing depth may not be the best way to increase performance, particularly given other limitations. Investigations into deep residual networks have also suggested that they may not in fact be operating as a single deep network, but rather as an ensemble of many relatively shallow networks. We examine these issues, and in doing so arrive at a new interpretation of the unravelled view of deep residual networks which explains some of the behaviours that have been observed experimentally. As a result, we are able to derive a new, shallower, architecture of residual networks which significantly outperforms much deeper models such as ResNet-200 on the ImageNet classification dataset. We also show that this performance is transferable to other problem domains by developing a semantic segmentation approach which outperforms the state-of-the-art by a remarkable margin on datasets including PASCAL VOC, PASCAL Context, and Cityscapes. The architecture that we propose thus outperforms its comparators, including very deep ResNets, and yet is more efficient in memory use and sometimes also in training time. The code and models are available at https://github.com/itijyou/ademxapp

Motivation & Objective

• Explain the unravelled view of deep residual networks and effective depth.
• Propose and evaluate shallower, wider residual architectures that outperform deeper counterparts.
• Demonstrate transferability of the proposed architectures to semantic segmentation benchmarks.
• Assess memory and training efficiency of the proposed networks relative to very deep ResNets.

Proposed method

• Provide an unravelled, effective-depth analysis of residual units with shortcut connections.
• Design a family of relatively shallow networks with two 3x3 convolutions per residual unit and selective bottlenecks.
• Evaluate on ImageNet to compare top-1/top-5 accuracy and throughput against deep ResNets and Inception variants.
• Adapt the classification networks for semantic segmentation by modifying down-sampling, dilations, and classifier structure without heavy multi-scale supervision.
• Train and fine-tune using MXNet on multi-GPU setups and report memory usage and training speed.

Experimental results

Research questions

• RQ1 Do residual networks operate as an exponential ensemble of shallow sub-networks, or a linearly growing ensemble?
• RQ2 Can shallower, wider residual architectures outperform deeper ResNets on ImageNet while being memory-efficient?
• RQ3 How well do the proposed architectures transfer to semantic segmentation benchmarks (PASCAL VOC, Cityscapes, ADE20K) without heavy post-processing?

Key findings

• Shallow, wide residual architectures can outperform very deep ResNets (e.g., ResNet-152, ResNet-200) on ImageNet in terms of top-1/top-5 accuracy.
• A network with around seventeen residual units can outperform deeper models while being more memory-efficient.
• Semantic segmentation using features from the proposed networks achieves state-of-the-art-like results on PASCAL VOC, Cityscapes, and ADE20K without multi-scale or CRF post-processing.
• Memory usage and training speed can improve with the shallower architecture, depending on input size and down-sampling strategy.
• Performance correlates with proper depth design and avoiding over-deepening, supporting a width-versus-depth trade-off that favors end-to-end trainability.