[Paper Review] DropBlock: A regularization method for convolutional networks

Golnaz Ghiasi, Tsung-Yi Lin+1 more|arXiv (Cornell University)|Oct 30, 2018

Advanced Neural Network ApplicationsComputer Science517 citations

TL;DR

DropBlock introduces structured dropout by dropping contiguous regions in feature maps, improving regularization and accuracy for CNNs compared to standard dropout. It yields notable gains on ImageNet and COCO across ResNet-50 and AmoebaNet architectures.

ABSTRACT

Deep neural networks often work well when they are over-parameterized and trained with a massive amount of noise and regularization, such as weight decay and dropout. Although dropout is widely used as a regularization technique for fully connected layers, it is often less effective for convolutional layers. This lack of success of dropout for convolutional layers is perhaps due to the fact that activation units in convolutional layers are spatially correlated so information can still flow through convolutional networks despite dropout. Thus a structured form of dropout is needed to regularize convolutional networks. In this paper, we introduce DropBlock, a form of structured dropout, where units in a contiguous region of a feature map are dropped together. We found that applying DropbBlock in skip connections in addition to the convolution layers increases the accuracy. Also, gradually increasing number of dropped units during training leads to better accuracy and more robust to hyperparameter choices. Extensive experiments show that DropBlock works better than dropout in regularizing convolutional networks. On ImageNet classification, ResNet-50 architecture with DropBlock achieves $78.13\%$ accuracy, which is more than $1.6\%$ improvement on the baseline. On COCO detection, DropBlock improves Average Precision of RetinaNet from $36.8\%$ to $38.4\%$.

Motivation & Objective

• Motivate the need for better regularization in convolutional networks due to spatially correlated features under standard dropout.
• Propose DropBlock as a structured dropout that drops contiguous regions in feature maps.
• Investigate the impact of applying DropBlock to convolutional blocks and skip connections, with scheduling of drop probability.
• Demonstrate effectiveness of DropBlock on ImageNet classification, COCO object detection, and PASCAL VOC semantic segmentation.

Proposed method

• Define DropBlock as dropping contiguous regions (blocks) within feature maps.
• Sample a Bernoulli mask per feature map to select seed locations, then expand each seed to a block of size block_size.
• Normalize the feature map after masking to maintain expected activation statistics.
• Introduce a scheduling approach that gradually decreases keep_prob during training for robustness.
• Evaluate DropBlock across multiple architectures (ResNet-50, AmoebaNet-B) and tasks (image classification, object detection, segmentation).

Experimental results

Research questions

• RQ1 Does structured dropout (DropBlock) outperform conventional dropout in CNN regularization?
• RQ2 What block sizes and scheduling strategies yield the best performance across architectures and tasks?
• RQ3 Is applying DropBlock to skip connections beneficial in residual networks?
• RQ4 Can DropBlock improve performance on diverse vision tasks beyond image classification (e.g., object detection and segmentation)?

Key findings

• DropBlock consistently outperforms dropout and SpatialDropout on ImageNet with ResNet-50, achieving 78.13% top-1 accuracy (kp=0.9) versus 76.51% baseline.
• DropBlock with scheduling and applying to skip connections yields further gains, reaching 78.35% top-1 with label smoothing.
• On AmoebaNet-B (6, 256), DropBlock improves top-1 accuracy from 82.25% to 82.52%.
• In COCO object detection with RetinaNet, DropBlock increases AP from 36.8% to 38.4%.
• DropBlock applied to semantic segmentation (PASCAL VOC) improves mIOU when trained from scratch, e.g., 53.4% with block_size=16 vs 47.2% without DropBlock.