[Paper Review] Qualitative Analysis of Monte Carlo Dropout

Ronald Seoh|arXiv (Cornell University)|Jul 3, 2020

Gaussian Processes and Bayesian Inference16 references30 citations

TL;DR

The paper analyzes Monte Carlo (MC) dropout as a practical variational inference method for Bayesian neural networks, detailing its uncertainty estimates, benefits, and costs through qualitative and empirical experiments. It highlights how MC dropout captures epistemic uncertainty, discusses hyperparameters, training requirements, and potential extensions to aleatoric uncertainty.

ABSTRACT

In this report, we present qualitative analysis of Monte Carlo (MC) dropout method for measuring model uncertainty in neural network (NN) models. We first consider the sources of uncertainty in NNs, and briefly review Bayesian Neural Networks (BNN), the group of Bayesian approaches to tackle uncertainties in NNs. After presenting mathematical formulation of MC dropout, we proceed to suggesting potential benefits and associated costs for using MC dropout in typical NN models, with the results from our experiments.

Motivation & Objective

• Motivate the need to quantify uncertainty in neural networks for high-stakes decisions.
• Explain MC dropout as a variational inference approach and its GP connection.
• Assess practical benefits and costs of MC dropout through qualitative and empirical analyses.
• Discuss hyperparameters, training requirements, and potential extensions to model aleatoric uncertainty.

Proposed method

• Describe sources of uncertainty in neural networks (epistemic vs. aleatoric).
• Present MC dropout formulation as a variational Bayesian approach with W_i = M_i · diag([Z_i,j]) and Z_i,j ~ Bernoulli(p_i).
• Derive predictive posterior via Monte Carlo averaging over T stochastic forward passes (Equation 10).
• Relate MC dropout to Gaussian Process approximation and discuss hyperparameters (p, τ, l).
• Conduct toy and real-data experiments to visualize uncertainty and compare with standard dropout (Figures 1–4, 5–9).
• Explore extension to heteroscedastic (input-dependent) aleatoric uncertainty (Equation 14).

Experimental results

Research questions

• RQ1 What uncertainty information does MC dropout capture in neural networks?
• RQ2 How do dropout rate, model precision, and training epochs affect predictive uncertainty and performance?
• RQ3 What are the practical costs and benefits of using MC dropout in real datasets?
• RQ4 Can MC dropout be extended to model heteroscedastic aleatoric uncertainty?
• RQ5 How many test-time predictions are needed for stable uncertainty estimates?

Key findings

• MC dropout provides uncertainty estimates by averaging predictions over multiple stochastic forward passes, approximating a Bayesian posterior.
• Higher dropout rates increase predictive uncertainty and can lead to poorer fit if over-regularizing, while model precision τ modulates added homoscedastic noise.
• More training epochs improve the quality of uncertainty estimates, with very few epochs yielding underfitting of uncertainty patterns.
• MC dropout can improve predictive performance on some regression tasks (e.g., yacht dataset) and reduce score variance across splits, though improvements over standard dropout are not universal across all datasets (Boston Housing, CIFAR-10 CNNs).
• A small number of test predictions (around 50) often suffices for stable uncertainty estimates in common settings; more predictions provide diminishing returns.
• Extending MC dropout to heteroscedastic aleatoric uncertainty (τ(x)) can capture total predictive uncertainty, though disentangling epistemic vs. aleatoric remains nontrivial.