[论文解读] Qualitative Analysis of Monte Carlo Dropout

Ronald Seoh|arXiv (Cornell University)|Jul 3, 2020

Gaussian Processes and Bayesian Inference参考文献 16被引用 30

一句话总结

本论文将蒙特卡洛 dropout 作为贝叶斯神经网络的实用变分推断方法进行分析，通过定性与经验性实验详细说明其不确定性估计、优点与成本。本文强调 MC dropout 如何捕捉认知性不确定性，讨论超参数、训练需求，以及对可别再扩展到对偶发性不确定性的潜在扩展。

ABSTRACT

In this report, we present qualitative analysis of Monte Carlo (MC) dropout method for measuring model uncertainty in neural network (NN) models. We first consider the sources of uncertainty in NNs, and briefly review Bayesian Neural Networks (BNN), the group of Bayesian approaches to tackle uncertainties in NNs. After presenting mathematical formulation of MC dropout, we proceed to suggesting potential benefits and associated costs for using MC dropout in typical NN models, with the results from our experiments.

研究动机与目标

• Motivate the need to quantify uncertainty in neural networks for high-stakes decisions.
• Explain MC dropout as a variational inference approach and its GP connection.
• Assess practical benefits and costs of MC dropout through qualitative and empirical analyses.
• Discuss hyperparameters, training requirements, and potential extensions to model aleatoric uncertainty.

提出的方法

• Describe sources of uncertainty in neural networks (epistemic vs. aleatoric).
• Present MC dropout formulation as a variational Bayesian approach with W_i = M_i · diag([Z_i,j]) and Z_i,j ~ Bernoulli(p_i).
• Derive predictive posterior via Monte Carlo averaging over T stochastic forward passes (Equation 10).
• Relate MC dropout to Gaussian Process approximation and discuss hyperparameters (p, τ, l).
• Conduct toy and real-data experiments to visualize uncertainty and compare with standard dropout (Figures 1–4, 5–9).
• Explore extension to heteroscedastic (input-dependent) aleatoric uncertainty (Equation 14).

实验结果

研究问题

• RQ1 What uncertainty information does MC dropout capture in neural networks?
• RQ2 How do dropout rate, model precision, and training epochs affect predictive uncertainty and performance?
• RQ3 What are the practical costs and benefits of using MC dropout in real datasets?
• RQ4 Can MC dropout be extended to model heteroscedastic aleatoric uncertainty?
• RQ5 How many test-time predictions are needed for stable uncertainty estimates?

主要发现

Size	RMSE (Dropout-Ron)	RMSE (Dropout-Gal)	LL (Dropout-Ron)	LL (Dropout-Gal)
yacht	2.82 B1 0.20	0.67 B1 0.05	-2.32 B1 0.07	-1.25 B1 0.01
bostonHousing	3.08 B1 0.19	2.90 B1 0.18	-2.88 B1 0.11	-2.40 B1 0.04
power-plant	4.19 B1 0.03	4.01 B1 0.04	-2.84 B1 0.01	-2.80 B1 0.01
naval-propulsion-plant	0.01 B1 0.00	0.00 B1 0.00	3.46 B1 0.01	4.45 B1 0.00

• MC dropout provides uncertainty estimates by averaging predictions over multiple stochastic forward passes, approximating a Bayesian posterior.
• Higher dropout rates increase predictive uncertainty and can lead to poorer fit if over-regularizing, while model precision τ modulates added homoscedastic noise.
• More training epochs improve the quality of uncertainty estimates, with very few epochs yielding underfitting of uncertainty patterns.
• MC dropout can improve predictive performance on some regression tasks (e.g., yacht dataset) and reduce score variance across splits, though improvements over standard dropout are not universal across all datasets (Boston Housing, CIFAR-10 CNNs).
• A small number of test predictions (around 50) often suffices for stable uncertainty estimates in common settings; more predictions provide diminishing returns.
• Extending MC dropout to heteroscedastic aleatoric uncertainty (τ(x)) can capture total predictive uncertainty, though disentangling epistemic vs. aleatoric remains nontrivial.