[Paper Review] EfficientNetV2: Smaller Models and Faster Training

Mingxing Tan, Quoc V. Le|arXiv (Cornell University)|Apr 1, 2021

Advanced Neural Network Applications41 references1,120 citations

TL;DR

Introduces EfficientNetV2, a family of smaller, faster-converging ConvNets found via training-aware NAS and progressive learning with adaptive regularization, achieving higher accuracy with fewer parameters and faster training than prior models.

ABSTRACT

This paper introduces EfficientNetV2, a new family of convolutional networks that have faster training speed and better parameter efficiency than previous models. To develop this family of models, we use a combination of training-aware neural architecture search and scaling, to jointly optimize training speed and parameter efficiency. The models were searched from the search space enriched with new ops such as Fused-MBConv. Our experiments show that EfficientNetV2 models train much faster than state-of-the-art models while being up to 6.8x smaller. Our training can be further sped up by progressively increasing the image size during training, but it often causes a drop in accuracy. To compensate for this accuracy drop, we propose to adaptively adjust regularization (e.g., dropout and data augmentation) as well, such that we can achieve both fast training and good accuracy. With progressive learning, our EfficientNetV2 significantly outperforms previous models on ImageNet and CIFAR/Cars/Flowers datasets. By pretraining on the same ImageNet21k, our EfficientNetV2 achieves 87.3% top-1 accuracy on ImageNet ILSVRC2012, outperforming the recent ViT by 2.0% accuracy while training 5x-11x faster using the same computing resources. Code will be available at https://github.com/google/automl/tree/master/efficientnetv2.

Motivation & Objective

• Motivate and improve training efficiency alongside parameter efficiency in ConvNets.
• Explore bottlenecks in EfficientNet training and identify architectural choices to accelerate training.
• Develop a training-aware NAS and scaling framework to jointly optimize accuracy, speed, and parameter count.
• Propose progressive learning with adaptive regularization to maintain accuracy while increasing image size during training.
• Demonstrate strong ImageNet and transfer-learning performance with efficient pretraining on ImageNet21k.

Proposed method

• Analyze training bottlenecks in EfficientNet (V1) and identify improvements.
• Extend the search space with MBConv and Fused-MBConv blocks and perform training-aware NAS.
• Apply a non-uniform, stage-wise scaling strategy and cap maximum training image size.
• Introduce progressive learning with adaptive regularization to speed up training without accuracy loss.
• Pretrain on ImageNet21k and finetune on ImageNet ILSVRC2012 and downstream datasets to assess generalization.

Experimental results

Research questions

• RQ1 Can training-aware NAS jointly optimize accuracy, training speed, and parameter efficiency for ConvNets?
• RQ2 Do MBConv and Fused-MBConv blocks, along with non-uniform scaling, yield faster training and smaller models without sacrificing accuracy?
• RQ3 Does progressive learning with adaptive regularization improve training speed while maintaining or boosting accuracy on ImageNet and transfer tasks?
• RQ4 How does EfficientNetV2 compare to prior ConvNets and ViTs in terms of training speed, parameter efficiency, and inference latency?
• RQ5 What is the impact of ImageNet21k pretraining on downstream transfer learning performance?

Key findings

• EfficientNetV2 models train 5x–11x faster and are up to 6.8x smaller in parameters than prior models on ImageNet.
• A training-aware NAS with a space including MBConv and Fused-MBConv yields EfficientNetV2 architectures that outperform EfficientNet on training speed and parameter efficiency.
• Progressive learning with adaptive regularization significantly speeds up training and can improve or maintain accuracy across ImageNet and transfer datasets.
• EfficientNetV2-M achieves comparable accuracy to EfficientNet-B7 while being trained 11x faster using the same resources.
• With ImageNet21k pretraining, EfficientNetV2-L (21k) reaches 87.3% top-1 on ImageNet and outperforms ViT-L/16(21k) while training 5x–11x faster.
• EfficientNetV2 shows strong transfer learning performance on CIFAR-10, CIFAR-100, Flowers, and Cars compared to prior ConvNets and ViTs.