[論文レビュー] Striving for Simplicity: The All Convolutional Net

Jost Tobias Springenberg, Alexey Dosovitskiy+2 more|arXiv (Cornell University)|Dec 21, 2014

Advanced Neural Network Applications参考文献 24被引用数 2,592

ひとこと要約

本論文は pooling を stride の畳み込みに置換することで性能が維持または向上すること、全畳み込みネットワーク（プーリングなし）が CIFAR-10/100 で最先端の結果を達成し、ImageNet でも競争力があること、そして新しい deconvolution ベースの可視化アプローチを提案していることを示している。

ABSTRACT

Most modern convolutional neural networks (CNNs) used for object recognition are built using the same principles: Alternating convolution and max-pooling layers followed by a small number of fully connected layers. We re-evaluate the state of the art for object recognition from small images with convolutional networks, questioning the necessity of different components in the pipeline. We find that max-pooling can simply be replaced by a convolutional layer with increased stride without loss in accuracy on several image recognition benchmarks. Following this finding -- and building on other recent work for finding simple network structures -- we propose a new architecture that consists solely of convolutional layers and yields competitive or state of the art performance on several object recognition datasets (CIFAR-10, CIFAR-100, ImageNet). To analyze the network we introduce a new variant of the "deconvolution approach" for visualizing features learned by CNNs, which can be applied to a broader range of network structures than existing approaches.

研究の動機と目的

• Question the necessity of max-pooling and other architectural components in CNNs for object recognition on small images.
• Propose an architecture composed solely of convolutional layers with strided downsampling.
• Evaluate the all-convolutional network on CIFAR-10, CIFAR-100, and ImageNet-scale data.
• Introduce a deconvolution-based visualization method suitable for networks without pooling.

提案手法

• Replace pooling layers with convolutional layers having stride two to achieve downsampling.
• Use small kernel sizes (primarily 3x3) to build deep, all-convolutional networks.
• Replace fully connected layers with 1x1 convolutions followed by global averaging and softmax for prediction.
• Compare three variants derived from base models to isolate the effect of pooling: Strided-CNN (increased stride), ConvPool-CNN (pooling replaced by conv), All-CNN (no pooling).
• Employ SGD with momentum, dropout, and weight decay, with data augmentation (horizontal flips, translations) for CIFAR-10/100 experiments.
• Deconvolution-based visualization: propose guided backpropagation to visualize high-layer features without dependence on pooling switches.

実験結果

リサーチクエスチョン

• RQ1 Is max-pooling necessary for competitive CNN performance on small-scale datasets?
• RQ2 Can an architecture built solely from convolutional layers (with strided downsampling) match or exceed state-of-the-art results on CIFAR-10/100?
• RQ3 How does removing pooling affect feature representations and visualization?
• RQ4 Do all-convolutional networks scale to larger datasets like ImageNet?
• RQ5 Can a deconvolution-based visualization approach be effectively applied to networks without pooling?

主な発見

• All-CNN architectures achieve state-of-the-art or competitive results on CIFAR-10/100 without max-pooling.
• Replacing pooling with strided convolution maintains or improves accuracy across variants and matches ConvPool-CNN performance in many cases.
• Small 3x3 convolutions stacked with occasional stride-2 downsampling outperform several prior architectures on CIFAR-10/100, sometimes with fewer parameters.
• On ImageNet-scale data, an upscaled All-CNN-B provided competitive results with far fewer parameters than AlexNet-level models, indicating pooling may be unnecessary for large networks as well.
• The proposed guided backpropagation visualization yields clearer feature visualizations for higher layers in networks without pooling compared to deconvnet methods that rely on switches.