[Paper Review] Dropout Training as Adaptive Regularization

Stefan Wager, Sida Wang+1 more|arXiv (Cornell University)|Jul 4, 2013

Machine Learning and Data Classification23 references273 citations

TL;DR

This paper frames dropout training as adaptive regularization in generalized linear models, showing it is first-order equivalent to L2 regularization after scaling features by the inverse diagonal Fisher information matrix. It establishes a connection to AdaGrad and proposes a semi-supervised method that uses unlabeled data to improve the regularizer, achieving state-of-the-art performance on the IMDB reviews dataset.

ABSTRACT

Dropout and other feature noising schemes control overfitting by artificially corrupting the training data. For generalized linear models, dropout performs a form of adaptive regularization. Using this viewpoint, we show that the dropout regularizer is first-order equivalent to an L2 regularizer applied after scaling the features by an estimate of the inverse diagonal Fisher information matrix. We also establish a connection to AdaGrad, an online learning algorithm, and find that a close relative of AdaGrad operates by repeatedly solving linear dropout-regularized problems. By casting dropout as regularization, we develop a natural semi-supervised algorithm that uses unlabeled data to create a better adaptive regularizer. We apply this idea to document classification tasks, and show that it consistently boosts the performance of dropout training, improving on state-of-the-art results on the IMDB reviews dataset.

Motivation & Objective

• To understand the theoretical mechanism behind dropout's success in controlling overfitting.
• To formalize dropout as a form of adaptive regularization in generalized linear models.
• To establish a connection between dropout training and the AdaGrad optimization algorithm.
• To develop a semi-supervised learning method that improves the dropout regularizer using unlabeled data.
• To demonstrate empirically that the proposed method outperforms existing state-of-the-art on benchmark text classification datasets.

Proposed method

• The paper derives that dropout training is first-order equivalent to L2 regularization applied after transforming features by the inverse square root of the estimated diagonal Fisher information matrix.
• It shows that the dropout regularizer favors rare but useful features in logistic regression, similar to adaptive regularization.
• A close relative of the AdaGrad algorithm is identified as solving linearized, dropout-regularized problems at each iteration.
• A semi-supervised regularizer is constructed by combining labeled and unlabeled data, where the unlabeled data informs the regularization strength through an estimated penalty function.
• The method uses a quadratic surrogate of the regularizer for optimization and applies cross-validation to tune the discount factor for unlabeled data.
• The approach is fully discriminative and does not require fitting a generative model.

Experimental results

Research questions

• RQ1 How does dropout training relate to regularization in generalized linear models?
• RQ2 Can dropout be interpreted as a form of adaptive L2 regularization, and if so, how is the adaptive scaling determined?
• RQ3 What is the connection between dropout training and the AdaGrad optimization algorithm?
• RQ4 Can unlabeled data be used to improve the regularization in dropout training, and if so, how?
• RQ5 Does the proposed semi-supervised method lead to improved performance on standard benchmark datasets?

Key findings

• Dropout training is first-order equivalent to L2 regularization after scaling features by the inverse square root of the estimated diagonal Fisher information matrix.
• The method improves generalization by adaptively regularizing features based on their statistical properties, favoring rare but useful features in logistic regression.
• A close relative of AdaGrad is shown to solve linearized, dropout-regularized problems at each step, revealing a deep connection between the two.
• Using unlabeled data to refine the regularizer consistently improves performance across multiple document classification tasks.
• On the IMDB reviews dataset, the semi-supervised dropout method achieves state-of-the-art accuracy, outperforming previous methods.
• The improvement is observed even with a large amount of labeled data, indicating that unlabeled data contributes meaningful inductive bias to the regularizer.