[Paper Review] Revisiting ResNets: Improved Training and Scaling Strategies

Irwan Bello, William Fedus+6 more|arXiv (Cornell University)|Mar 13, 2021

Advanced Neural Network Applications69 references209 citations

TL;DR

The paper disentangles architecture, training, and scaling, showing improved training and scaling strategies yield ResNets that rival state-of-the-art models, with ResNet-RS achieving substantial speedups over EfficientNets while maintaining accuracy.

ABSTRACT

Novel computer vision architectures monopolize the spotlight, but the impact of the model architecture is often conflated with simultaneous changes to training methodology and scaling strategies. Our work revisits the canonical ResNet (He et al., 2015) and studies these three aspects in an effort to disentangle them. Perhaps surprisingly, we find that training and scaling strategies may matter more than architectural changes, and further, that the resulting ResNets match recent state-of-the-art models. We show that the best performing scaling strategy depends on the training regime and offer two new scaling strategies: (1) scale model depth in regimes where overfitting can occur (width scaling is preferable otherwise); (2) increase image resolution more slowly than previously recommended (Tan & Le, 2019). Using improved training and scaling strategies, we design a family of ResNet architectures, ResNet-RS, which are 1.7x - 2.7x faster than EfficientNets on TPUs, while achieving similar accuracies on ImageNet. In a large-scale semi-supervised learning setup, ResNet-RS achieves 86.2% top-1 ImageNet accuracy, while being 4.7x faster than EfficientNet NoisyStudent. The training techniques improve transfer performance on a suite of downstream tasks (rivaling state-of-the-art self-supervised algorithms) and extend to video classification on Kinetics-400. We recommend practitioners use these simple revised ResNets as baselines for future research.

Motivation & Objective

• Assess the impact of training methods on ResNet performance independent of architectural changes.
• Systematically study scaling strategies (depth, width, resolution) across training regimes to derive best practices.
• Develop ResNet-RS families that optimize speed-accuracy on TPUs/GPUs.
• Evaluate semi-supervised and transfer learning performance of the updated ResNets.
• Demonstrate generality of improved training/scaling strategies to related tasks (video, downstream tasks).

Proposed method

• Apply modern training and regularization techniques to the canonical ResNet architecture.
• Perform additive ablation studies to quantify effects of training methods, regularization, and architectural tweaks.
• Systematically scale models across depth, width, and image resolution with full training duration (350 epochs).
• Compare scaling strategies under different training regimes to identify depth vs width trade-offs.
• Design ResNet-RS by applying the improved training and scaling strategies across a range of model sizes.
• Evaluate semi-supervised performance with pseudo-labeled data and transfer learning on downstream tasks.
• Analyze latency, memory, and FLOPs to explain speedups over EfficientNets.

Experimental results

Research questions

• RQ1 How do training methods and regularization interact to affect ResNet performance independent of architecture changes?
• RQ2 What scaling strategy (depth vs width vs resolution) yields the best speed-accuracy Pareto under different training regimes?
• RQ3 Can revised ResNets (ResNet-RS) outperform EfficientNets on TPUs/GPUs while maintaining comparable accuracy?
• RQ4 Do improved training and scaling strategies transfer to semi-supervised learning and downstream tasks (including video)?

Key findings

• Improved training methods alone raise ResNet-200 top-1 accuracy from 79.0% to 82.2%.
• Adding SE and ResNet-D raises top-1 to 83.4% (with training methods contributing most of the gain).
• ResNet-RS models are 1.7x–2.7x faster than EfficientNets on TPUs (2.1x–3.3x on GPUs) for similar accuracy on ImageNet.
• In semi-supervised learning with 130M pseudo-labeled images, ResNet-RS-152 achieves 86.2% top-1 (4.7x faster on TPU than EfficientNet-NoisyStudent).
• ResNet-RS improves 3D-ResNet on Kinetics-400 by +4.0% (from 73.4% to 77.4%).
• ResNet checkpoints with improved training rival or outperform self-supervised representations on downstream tasks (comparable to SimCLR variants).
• EfficientNet-RS (slow image scaling) improves Pareto efficiency, but ResNet-RS still edges out on speed-accuracy curves.