[Paper Review] Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Yarin Gal, Zoubin Ghahramani|arXiv (Cornell University)|Jun 6, 2015

Gaussian Processes and Bayesian Inference36 references4,095 citations

TL;DR

The paper shows that dropout in deep neural networks approximates Bayesian inference in deep Gaussian processes, enabling practical estimation of model uncertainty and improving predictive likelihood and RMSE on regression, classification, and reinforcement learning tasks.

ABSTRACT

Deep learning tools have gained tremendous attention in applied machine learning. However such tools for regression and classification do not capture model uncertainty. In comparison, Bayesian models offer a mathematically grounded framework to reason about model uncertainty, but usually come with a prohibitive computational cost. In this paper we develop a new theoretical framework casting dropout training in deep neural networks (NNs) as approximate Bayesian inference in deep Gaussian processes. A direct result of this theory gives us tools to model uncertainty with dropout NNs -- extracting information from existing models that has been thrown away so far. This mitigates the problem of representing uncertainty in deep learning without sacrificing either computational complexity or test accuracy. We perform an extensive study of the properties of dropout's uncertainty. Various network architectures and non-linearities are assessed on tasks of regression and classification, using MNIST as an example. We show a considerable improvement in predictive log-likelihood and RMSE compared to existing state-of-the-art methods, and finish by using dropout's uncertainty in deep reinforcement learning.

Motivation & Objective

• Motivate the need for representing model uncertainty in deep learning for regression, classification, and reinforcement learning.
• Show that dropout can be interpreted as a Bayesian approximation to deep Gaussian processes.
• Develop practical tools to extract and utilize uncertainty from existing dropout networks.
• Evaluate uncertainty estimates across architectures and nonlinearities on regression and MNIST classification.

Proposed method

• Cast dropout training as approximate Bayesian inference in deep Gaussian processes.
• Use a variational distribution over sparse spectral components with Bernoulli dropout masks to approximate the posterior.
• Formulate a KL-divergence objective that relates dropout to a GP-based posterior (Eq. 3 and 4 in the text).
• Derive Monte Carlo dropout (MC dropout) by sampling dropout masks to estimate predictive mean and variance (Eq. 6 and 7).
• Provide practical procedures to compute predictive log-likelihood and uncertainty via MC forward passes.

Experimental results

Research questions

• RQ1 Can dropout be interpreted as a Bayesian approximation for representing uncertainty in deep models?
• RQ2 How can we quantify predictive uncertainty from standard dropout networks without changing architecture or training procedures?
• RQ3 Do uncertainty estimates from MC dropout improve predictive log-likelihood and RMSE across tasks (regression, classification, RL) compared to other methods?
• RQ4 How does dropout-derived uncertainty behave across different architectures and nonlinearities (ReLU, TanH, convnets) and data regimes?

Key findings

• Dropout NNs can be viewed as approximate inference in deep Gaussian processes, enabling uncertainty quantification without extra computational burden.
• MC dropout provides practical estimates of predictive mean and variance by averaging over multiple stochastic forward passes.
• Uncertainty from dropout improves predictive log-likelihood and RMSE on regression benchmarks (e.g., MNIST-based classification and standard datasets) relative to several baselines.
• Uncertainty information from dropout can be leveraged in reinforcement learning to guide exploration (e.g., Thompson sampling).
• Across architectures and nonlinearities, dropout-derived uncertainty is especially informative for classification tasks and extrapolation scenarios.