[Paper Review] Dropout with Expectation-linear Regularization

Xuezhe Ma, Yingkai Gao+4 more|arXiv (Cornell University)|Sep 26, 2016

Adversarial Robustness in Machine Learning26 references13 citations

TL;DR

This paper proposes expectation-linear regularization to explicitly control the inference gap in dropout training, where the model's test-time behavior deviates from its training-time ensemble approximation. By regularizing the difference between expected outputs during training and inference, the method improves generalization with no computational overhead, achieving consistent accuracy gains on MNIST, CIFAR-10, and CIFAR-100 across multiple architectures.

ABSTRACT

Dropout, a simple and effective way to train deep neural networks, has led to a number of impressive empirical successes and spawned many recent theoretical investigations. However, the gap between dropout's training and inference phases, introduced due to tractability considerations, has largely remained under-appreciated. In this work, we first formulate dropout as a tractable approximation of some latent variable model, leading to a clean view of parameter sharing and enabling further theoretical analysis. Then, we introduce (approximate) expectation-linear dropout neural networks, whose inference gap we are able to formally characterize. Algorithmically, we show that our proposed measure of the inference gap can be used to regularize the standard dropout training objective, resulting in an \emph{explicit} control of the gap. Our method is as simple and efficient as standard dropout. We further prove the upper bounds on the loss in accuracy due to expectation-linearization, describe classes of input distributions that expectation-linearize easily. Experiments on three image classification benchmark datasets demonstrate that reducing the inference gap can indeed improve the performance consistently.

Motivation & Objective

• To formally characterize the inference gap in standard dropout, where test-time inference uses a single scaled network instead of an ensemble.
• To develop a theoretically grounded, tractable method to quantify and control this gap during training.
• To propose a regularization scheme that explicitly minimizes the discrepancy between training and inference behaviors.
• To demonstrate that reducing the inference gap leads to improved model generalization without sacrificing training efficiency.
• To establish a connection between weight constraints (e.g., max-norm) and reduced inference gap through theoretical analysis.

Proposed method

• Formulates dropout as a tractable approximation of a latent variable model, enabling clearer analysis of parameter sharing.
• Introduces the concept of expectation-linearization to measure the deviation between expected outputs during training and inference.
• Proposes a regularized training objective that adds a sampled estimate of the inference gap to the standard dropout loss.
• Uses Monte Carlo sampling to approximate the expectation-linearization risk during training for optimization.
• Employs a hyperparameter λ to balance the trade-off between inference gap reduction and model accuracy.
• Theoretical analysis shows that max-norm constraints on weights reduce the inference gap, providing a justification for their empirical success.

Experimental results

Research questions

• RQ1 How can the inference gap in dropout—where test-time behavior differs from training-time ensemble averaging—be formally quantified?
• RQ2 What is the relationship between weight constraints like max-norm and the magnitude of the inference gap?
• RQ3 Can explicitly regularizing the inference gap lead to improved generalization performance in practice?
• RQ4 How does the trade-off between inference gap reduction and model accuracy behave under varying regularization strength?
• RQ5 Does the proposed method outperform or match existing dropout variants like Monte Carlo dropout and dropout distillation?

Key findings

• On MNIST, the proposed method reduced test error from 1.07% (standard dropout) to 0.90% (with expectation-linearization) on a 2-layer ReLU network with 8192 units.
• On CIFAR-10, the method achieved 10.86% error with AllConv, comparable to the 10.81% from dropout distillation and better than standard dropout (11.18%).
• On CIFAR-100, the method achieved 35.10% error with AllConv, outperforming standard dropout (35.50%) and matching dropout distillation (35.07%).
• The empirical expectation-linearization risk (Δ̂) decreased with increasing λ, confirming that higher regularization reduces the inference gap.
• The optimal λ value balances gap reduction and accuracy, with performance peaking around λ=1.0 to 2.0 across datasets.
• Theoretical analysis confirms that max-norm constraints reduce the inference gap, providing a justification for their widespread use in deep learning.