[논문 리뷰] VanillaNet: the Power of Minimalism in Deep Learning

Hanting Chen, Yunhe Wang외 2명|arXiv (Cornell University)|2023. 05. 22.

Advanced Neural Network Applications인용 수 83

한 줄 요약

VanillaNet은 단순하고 얕은 합성곱 아키텍처가 단축 경로나 셀프 어텐션 없이, 심층 학습 전략과 시리즈 활성화 함수를 사용하여도, 아키텍처 복잡도와 대기 시간(latency)을 크게 줄이면서도 최첨단 성능과 일치할 수 있음을 보여준다.

ABSTRACT

At the heart of foundation models is the philosophy of "more is different", exemplified by the astonishing success in computer vision and natural language processing. However, the challenges of optimization and inherent complexity of transformer models call for a paradigm shift towards simplicity. In this study, we introduce VanillaNet, a neural network architecture that embraces elegance in design. By avoiding high depth, shortcuts, and intricate operations like self-attention, VanillaNet is refreshingly concise yet remarkably powerful. Each layer is carefully crafted to be compact and straightforward, with nonlinear activation functions pruned after training to restore the original architecture. VanillaNet overcomes the challenges of inherent complexity, making it ideal for resource-constrained environments. Its easy-to-understand and highly simplified architecture opens new possibilities for efficient deployment. Extensive experimentation demonstrates that VanillaNet delivers performance on par with renowned deep neural networks and vision transformers, showcasing the power of minimalism in deep learning. This visionary journey of VanillaNet has significant potential to redefine the landscape and challenge the status quo of foundation model, setting a new path for elegant and effective model design. Pre-trained models and codes are available at https://github.com/huawei-noah/VanillaNet and https://gitee.com/mindspore/models/tree/master/research/cv/vanillanet.

연구 동기 및 목표

• 자원 제약 환경에서 배치 용이성을 높이기 위해 미니멀리스트 CNN 설계로의 전환을 촉진한다.
• 깊이, 단축 경로, 셀프 어텐션을 피하면서도 경쟁력 있는 성능을 유지하는 VanillaNet 아키텍처를 제안한다.
• 얕은 네트에서의 제한된 비선형성을 보상하기 위한 학습 및 활성화 기법을 개발한다.
• 대규모 이미지 분류 및 다운스트림 과제에서 VanillaNet을 평가하여 효율성-정확도 트레이드오프를 벤치마크한다.

제안 방법

• Introduce VanillaNet: stem, a single-layer-per-stage architecture with 4x4x3xC stem, stride-4 convolution, and subsequent stages using 1x1 convolutions with channel doubling (except final stage).
• Use a deep training strategy that gradually merges pairs of convolutions by replacing activations with a weighted identity mix A'(x)=(1-λ)A(x)+λx, with λ=e/E across epochs.
• Propose a series activation function A_s(x) = sum_{i=-n}^{n} a_i A(x + b_i) (and variants with neighbor-shift) to enhance non-linearity without heavy cost.
• Merge BN and adjacent convolutions post-training to obtain a single convolution for efficient inference (special handling for 1x1 convolutions).
• Implement a series-based activation to enable global information exchange across feature maps and compare its runtime cost to standard convolution (O(SA) ≪ O(CONV) in practical settings).
• Conduct ablations on the number of series terms n, deep-training, and the presence/location of shortcuts (none provide clear gains in VanillaNet).

실험 결과

연구 질문

• RQ1 Can a shallow, fully convolutional network without shortcuts or self-attention achieve competitive ImageNet accuracy?
• RQ2 Do deep-training and series activation techniques reliably raise the performance of minimalist VanillaNet variants?
• RQ3 What is the impact of removing shortcuts on performance and inference speed in minimalist architectures?
• RQ4 How does VanillaNet perform on downstream tasks (e.g., COCO) compared to state-of-the-art backbones?

주요 결과

• VanillaNet with series activation (n=3) attains 76.36% top-1 on ImageNet for VanillaNet-6 and 76.36% overall with deep training.
• Deep training plus series activation substantially improves vanilla shallow networks (e.g., AlexNet gains ~6%); ResNet-50 gains are marginal, indicating diminishing returns for already deep, non-minimal models.
• Shortcuts provide little or no accuracy gains for VanillaNet; may even slightly reduce non-linearity-driven performance in this minimal architecture.
• VanillaNet-9 achieves 79.87% top-1 with 2.91 ms latency on Nvidia A100 (batch size 1); VanillaNet-13-1.5× reaches 83.11% top-1 with 7.83 ms latency, indicating strong speed-accuracy trade-offs for shallow minimalist nets.
• On ImageNet, VanillaNet-9-13-1.5× shows competitive accuracy (up to ~83.1% real accuracy) with significantly different depth and latency profiles compared to ResNet-50 and ConvNext variants.
• In COCO, VanillaNet-13 delivers competitive AP metrics and higher FPS than some backbone Swin/ConvNext variants despite higher FLOPs/parameters, suggesting efficiency advantages in real-time settings