[论文解读] Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

Christian Szegedy, Sergey Ioffe+2 more|arXiv (Cornell University)|Feb 23, 2016

Domain Adaptation and Few-Shot Learning被引用 200

一句话总结

该论文评估了 Inception-v4、Inception-ResNet-v1/v2 及带残差连接的变体，结果表明残差可加速训练，且更大的模型能获得更高的准确性，集成方法在 ImageNet 上实现了最先进的前5名表现。

ABSTRACT

Very deep convolutional networks have been central to the largest advances in image recognition performance in recent years. One example is the Inception architecture that has been shown to achieve very good performance at relatively low computational cost. Recently, the introduction of residual connections in conjunction with a more traditional architecture has yielded state-of-the-art performance in the 2015 ILSVRC challenge; its performance was similar to the latest generation Inception-v3 network. This raises the question of whether there are any benefit in combining the Inception architecture with residual connections. Here we give clear empirical evidence that training with residual connections accelerates the training of Inception networks significantly. There is also some evidence of residual Inception networks outperforming similarly expensive Inception networks without residual connections by a thin margin. We also present several new streamlined architectures for both residual and non-residual Inception networks. These variations improve the single-frame recognition performance on the ILSVRC 2012 classification task significantly. We further demonstrate how proper activation scaling stabilizes the training of very wide residual Inception networks. With an ensemble of three residual and one Inception-v4, we achieve 3.08 percent top-5 error on the test set of the ImageNet classification (CLS) challenge

研究动机与目标

• Investigate whether adding residual connections to Inception architectures improves training speed and final accuracy.
• Develop streamlined Inception-based architectures (Inception-v4 and residual variants) with competitive performance.
• Assess training stability and scaling issues in very deep Inception-based networks.
• Evaluate single-model and ensemble performance on ImageNet (ILSVRC) to establish state-of-the-art results.

提出的方法

• Replace Inception's filter concatenation with residual connections in Inception blocks.
• Introduce Inception-v4 with a uniform, streamlined architecture of more inception modules.
• Use filter-expansion layers to match dimensionality before residual additions in Inception-ResNet variants.
• Scale residuals to stabilize training when network width becomes very large.
• Train with TensorFlow on 20 replicas using RMSProp with decay 0.9 and epsilon 1.0; learning rate 0.045 with exponential decay 0.94 every two epochs.

实验结果

研究问题

• RQ1 Do residual connections accelerate training of Inception-based networks compared to pure Inception variants of similar cost?
• RQ2 Can Inception-ResNet variants outperform non-residual Inception models at similar computational cost?
• RQ3 What is the impact of model size and ensemble methods on ImageNet top-5 accuracy, and does ensemble achieve state-of-the-art results on the validation/test sets?

主要发现

• Residual Inception networks train faster than pure Inception equivalents of similar cost.
• Residual variants achieve slightly better final accuracy than non-residual counterparts at similar cost in some configurations.
• Inception-ResNet-v2 and Inception-v4 deliver leading single-model top-5 error among their peers, with 3.1% top-5 error reported for an ensemble.
• An ensemble of Inception-v4 and three Inception-ResNet-v2 models achieves a 3.08% top-5 error on the ImageNet test set, representing a state-of-the-art at the time.
• Scaling residuals helps stabilize training when networks become very wide (residual scaling factors around 0.1–0.3).
• Increasing model size yields improved recognition performance across variants; ensemble gains surpass single-model gains in absolute improvements.