[Paper Review] Effective and Efficient Dropout for Deep Convolutional Neural Networks

Shaofeng Cai, Jinyang Gao+4 more|arXiv (Cornell University)|Apr 6, 2019

Advanced Neural Network Applications49 references55 citations

TL;DR

The paper analyzes dropout variants for CNNs, identifies BN conflicts with dropout, and proposes Drop-Conv2d and related building blocks (Drop-Neuron, Drop-Channel, Drop-Path) to achieve better regularization with minimal overhead across CNN architectures.

ABSTRACT

Convolutional Neural networks (CNNs) based applications have become ubiquitous, where proper regularization is greatly needed. To prevent large neural network models from overfitting, dropout has been widely used as an efficient regularization technique in practice. However, many recent works show that the standard dropout is ineffective or even detrimental to the training of CNNs. In this paper, we revisit this issue and examine various dropout variants in an attempt to improve existing dropout-based regularization techniques for CNNs. We attribute the failure of standard dropout to the conflict between the stochasticity of dropout and its following Batch Normalization (BN), and propose to reduce the conflict by placing dropout operations right before the convolutional operation instead of BN, or totally address this issue by replacing BN with Group Normalization (GN). We further introduce a structurally more suited dropout variant Drop-Conv2d, which provides more efficient and effective regularization for deep CNNs. These dropout variants can be readily integrated into the building blocks of CNNs and implemented in existing deep learning platforms. Extensive experiments on benchmark datasets including CIFAR, SVHN and ImageNet are conducted to compare the existing building blocks and the proposed ones with dropout training. Results show that our building blocks improve over state-of-the-art CNNs significantly, which is mainly due to the better regularization and implicit model ensemble effect.

Motivation & Objective

• Motivate robust regularization for deep CNNs to combat overfitting and improve generalization.
• Systematically analyze dropout variants (neuron, channel, path) in CNNs and their interaction with Batch Normalization and data augmentation.
• Develop unified convolutional building blocks that integrate dropout effectively and efficiently into common CNN architectures.
• Introduce Drop-Conv2d as a scalable, plug-and-play regularization technique that can be merged back during inference.
• Demonstrate extensive empirical gains on standard benchmarks (CIFAR, SVHN, ImageNet) using the proposed blocks.

Proposed method

• Formulate CNN transformations in a split-transform-aggregate framework focusing on channel-level operations.
• Compare dropout variants (Drop-Neuron, Drop-Channel, Drop-Path) and analyze their interaction with Batch Normalization and Group Normalization.
• Place dropout before convolution in the building blocks to reduce gradient variance and variance shift.
• Propose Drop-Conv2d by replicating each channel connection into P paths and applying dropout to these paths with re-aggregation at inference.
• Provide convolutional building blocks that integrate dropout (Drop-Neuron, Drop-Channel) for easy adoption in existing architectures.
• Evaluate the proposed blocks on CNN architectures across CIFAR, SVHN, and ImageNet to show performance gains.

Experimental results

Research questions

• RQ1 How does dropout interact with Batch Normalization in CNNs, and why can standard dropout be ineffective in convolutional blocks?
• RQ2 Are channel- and path-level dropouts (Drop-Channel, Drop-Path) more effective for CNN regularization than neuron-level dropout?
• RQ3 Can dropout be integrated into CNN blocks in a way that yields better regularization with minimal overhead?
• RQ4 Does Drop-Conv2d improve generalization across standard CNN architectures on benchmarks like CIFAR, SVHN, and ImageNet?

Key findings

• Drop-channel and drop-path generally outperform drop-neuron in CNN training due to better alignment with convolutional channel structure and BN interaction.
• Placing dropout right before convolutional layers reduces variance shift and stabilizes BN, improving training efficiency.
• Drop-Conv2d, which replicates channel connections into P paths and applies dropout, yields stronger regularization with negligible inference overhead.
• BN can introduce variance shift under dropout, which is mitigated by proper placement of dropout or by replacing BN with Group Normalization.
• The proposed building blocks produce significant accuracy improvements over state-of-the-art CNNs on CIFAR, SVHN, and ImageNet in experiments.