[Paper Review] Stochastic Pooling for Regularization of Deep Convolutional Neural Networks

Matthew D. Zeiler, Rob Fergus|arXiv (Cornell University)|Jan 16, 2013

Advanced Neural Network Applications7 references573 citations

TL;DR

This paper proposes stochastic pooling, a hyper-parameter-free regularization technique for deep convolutional neural networks that replaces deterministic max or average pooling with stochastic sampling from a multinomial distribution based on activation values within each pooling region. The method reduces overfitting and achieves state-of-the-art performance on multiple image datasets without data augmentation, outperforming conventional pooling and dropout in convolutional layers.

ABSTRACT

We introduce a simple and effective method for regularizing large convolutional neural networks. We replace the conventional deterministic pooling operations with a stochastic procedure, randomly picking the activation within each pooling region according to a multinomial distribution, given by the activities within the pooling region. The approach is hyper-parameter free and can be combined with other regularization approaches, such as dropout and data augmentation. We achieve state-of-the-art performance on four image datasets, relative to other approaches that do not utilize data augmentation.

Motivation & Objective

• To address overfitting in deep convolutional neural networks, particularly in convolutional layers where traditional regularization like dropout is less effective.
• To develop a regularization method that leverages existing network activations without discarding information, unlike dropout.
• To improve generalization performance on image classification tasks without relying on data augmentation.
• To create a simple, plug-and-play regularization technique with no hyper-parameters to tune.
• To demonstrate that stochastic pooling can achieve superior test accuracy compared to deterministic pooling and other regularization methods.

Proposed method

• Replace conventional deterministic pooling (max or average) with stochastic sampling from a multinomial distribution derived from the activations within each pooling region.
• Compute pooling region probabilities as normalized activations: $ p_i = \frac{a_i}{\sum_{k \in R_j} a_k} $, where $ a_i $ is the activation at position $ i $ in region $ j $.
• Sample the pooled activation from this multinomial distribution during training, introducing stochasticity that acts as regularization.
• Use the same stochastic sampling process during inference, but with a probability-weighted average of multiple forward passes to improve robustness.
• Combine stochastic pooling with other regularization techniques such as weight decay, dropout, and data augmentation.
• Apply deconvolutional networks to visualize feature maps by reconstructing input pixels using stochastically selected pooling locations, revealing structural consistency across samples.

Experimental results

Research questions

• RQ1 Can stochastic pooling effectively regularize deep convolutional networks and reduce overfitting in convolutional layers where dropout is less effective?
• RQ2 How does stochastic pooling compare to deterministic pooling (max and average) in terms of generalization performance on image classification tasks?
• RQ3 Does stochastic pooling achieve state-of-the-art performance when combined with other regularization techniques, especially without data augmentation?
• RQ4 Can the stochastic sampling process be interpreted as implicitly generating data augmentations through local deformations of feature maps?
• RQ5 To what extent do the multinomial probabilities in stochastic pooling encode meaningful structural information about the input data, as revealed by deconvolutional visualization?

Key findings

• Stochastic pooling achieves a test error of 15.12% on CIFAR-10 when trained with stochastic pooling and tested with probability weighting, outperforming max pooling (17.66%) and average pooling (53.50%).
• On the SVHN dataset, stochastic pooling achieves a test error of 15.20% when trained with stochastic pooling and tested with probability weighting, representing state-of-the-art performance among methods not using data augmentation.
• The method is hyper-parameter free and incurs negligible computational overhead, making it easily integrable into existing CNN architectures.
• Deconvolutional visualizations show that stochastic pooling preserves structural information in feature maps, especially when sampling from feed-forward probabilities rather than uniform distributions.
• Stochastic pooling combined with probability weighting at test time yields the best performance, indicating that ensembling multiple stochastic forward passes improves robustness.
• The approach is effective even when combined with other regularization techniques like weight decay and dropout, demonstrating compatibility and additive benefits.