[Paper Review] Improving Deep Neural Networks with Probabilistic Maxout Units

Jost Tobias Springenberg, Martin Riedmiller|arXiv (Cornell University)|Dec 20, 2013

Advanced Neural Network Applications11 references90 citations

TL;DR

This paper proposes probabilistic maxout units (probout) as a stochastic generalization of maxout units that improve invariance to input perturbations while preserving the favorable optimization and dropout generalization properties of maxout. The method uses a softmax-based probabilistic pooling mechanism over linear feature maps, achieving state-of-the-art classification accuracy of 2.39% on SVHN without data augmentation, matching or exceeding prior results on CIFAR-10 and CIFAR-100.

ABSTRACT

We present a probabilistic variant of the recently introduced maxout unit. The success of deep neural networks utilizing maxout can partly be attributed to favorable performance under dropout, when compared to rectified linear units. It however also depends on the fact that each maxout unit performs a pooling operation over a group of linear transformations and is thus partially invariant to changes in its input. Starting from this observation we ask the question: Can the desirable properties of maxout units be preserved while improving their invariance properties ? We argue that our probabilistic maxout (probout) units successfully achieve this balance. We quantitatively verify this claim and report classification performance matching or exceeding the current state of the art on three challenging image classification benchmarks (CIFAR-10, CIFAR-100 and SVHN).

Motivation & Objective

• To improve the invariance properties of maxout units, which are partially invariant due to subspace pooling, without sacrificing their favorable optimization and dropout generalization behavior.
• To develop a stochastic generalization of maxout that maintains piecewise linearity, avoids saturation, and preserves positive activation, while enhancing robustness to input transformations.
• To enable more even gradient flow across the k linear feature mappings within each unit, improving feature utilization in deep networks.
• To evaluate the proposed units on standard image classification benchmarks and demonstrate competitive or superior performance compared to state-of-the-art models.

Proposed method

• The probabilistic maxout unit replaces the deterministic max operation in standard maxout with a softmax-based probabilistic pooling over k linear feature maps, enabling stochastic activation.
• Each unit computes a probability distribution over its k linear transformations using a softmax function, allowing for stochastic sampling during forward and backward passes.
• The method maintains piecewise linearity and avoids saturation, preserving the optimization advantages of maxout while enhancing invariance to input perturbations.
• During backpropagation, gradients are distributed more evenly across all k feature maps, improving learning efficiency and feature utilization.
• The model is trained using standard backpropagation with cross-entropy loss and tested with model averaging over multiple forward passes to approximate dropout ensemble performance.
• The approach is evaluated on CIFAR-10, CIFAR-100, and SVHN using convolutional and fully connected layers with subspace pooling over k=5 or k=2 dimensions.

Experimental results

Research questions

• RQ1 Can a probabilistic generalization of maxout units improve invariance to input transformations while preserving the optimization and regularization benefits of dropout?
• RQ2 Does replacing the max operation with a probabilistic pooling mechanism lead to better gradient flow across the k linear feature maps in each unit?
• RQ3 Can the proposed probabilistic maxout units achieve state-of-the-art performance on standard image classification benchmarks without data augmentation?
• RQ4 How does the invariance of feature representations compare between maxout and probabilistic maxout networks under translation and rotation perturbations?

Key findings

• The probabilistic maxout model achieved a test error of 2.39% on the SVHN dataset without data augmentation, matching the current state of the art and outperforming the original maxout model (2.47%).
• On CIFAR-10, the model achieved a test error of 8.48%, matching the performance of the best-reported maxout model and surpassing other baselines such as ReLU with dropout.
• On CIFAR-100, the model achieved a test error of 27.68%, which is competitive with the state of the art and significantly better than several strong baselines.
• Feature-level analysis showed that probout networks exhibit significantly lower Euclidean distance between feature vectors of transformed and original images, indicating stronger invariance to translation and rotation.
• The mean distance across layers was consistently lower for probout than for maxout, particularly in the fully connected layer, demonstrating improved robustness in higher-level representations.
• The probabilistic formulation enabled more even gradient distribution across the k feature maps, leading to better utilization of the full k-dimensional subspace during training.