[Paper Review] Confidence Calibration for Convolutional Neural Networks Using Structured Dropout

Zhilu Zhang, Adrian V. Dalca+1 more|arXiv (Cornell University)|Jun 23, 2019

Adversarial Robustness in Machine Learning44 references41 citations

TL;DR

The paper treats MC dropout as an ensemble and shows structured dropout (dropBlock, dropChannel, dropLayer) increases model diversity and improves confidence calibration versus standard MC dropout and deep ensembles, with competitive accuracy.

ABSTRACT

In classification applications, we often want probabilistic predictions to reflect confidence or uncertainty. Dropout, a commonly used training technique, has recently been linked to Bayesian inference, yielding an efficient way to quantify uncertainty in neural network models. However, as previously demonstrated, confidence estimates computed with a naive implementation of dropout can be poorly calibrated, particularly when using convolutional networks. In this paper, through the lens of ensemble learning, we associate calibration error with the correlation between the models sampled with dropout. Motivated by this, we explore the use of structured dropout to promote model diversity and improve confidence calibration. We use the SVHN, CIFAR-10 and CIFAR-100 datasets to empirically compare model diversity and confidence errors obtained using various dropout techniques. We also show the merit of structured dropout in a Bayesian active learning application.

Motivation & Objective

• Motivate probabilistic predictions to reflect uncertainty in CNNs and address miscalibration with MC dropout.
• Frame dropout as ensemble sampling and link calibration error to model diversity.
• Propose structured dropout to promote diversity and improve calibration.
• Empirically compare structured dropout variants on SVHN, CIFAR-10, and CIFAR-100.
• Demonstrate practical benefits in Bayesian active learning.

Proposed method

• View MC dropout as an ensemble of dropout models and relate calibration to ensemble diversity via ambiguity and MSE decomposition.
• Introduce structured dropout variants: dropBlock (patch-level), dropChannel (channel-level), dropLayer (layer-level).
• Train with these dropouts and evaluate test-time MC samples (T=30) for uncertainty metrics (ECE, NLL, Brier) and accuracy.
• Compare against regular dropout and deep ensembles across SVHN, CIFAR-10, CIFAR-100 using PreAct-ResNet variants.
• Assess ensemble diversity with Interrater Agreement (IA) and study calibration via reliability diagrams.
• Apply structured dropout to Bayesian active learning to show improved data acquisition decisions.

Experimental results

Research questions

• RQ1 Can structured dropout increase model diversity in CNN ensembles compared to standard dropout?
• RQ2 Does increased diversity from structured dropout improve confidence calibration (ECE, NLL, Brier) without sacrificing accuracy?
• RQ3 How do structured dropout variants perform relative to deep ensembles on calibration and uncertainty metrics?
• RQ4 Are there practical benefits of structured dropout in active learning scenarios?

Key findings

• MC dropChannel and MC dropLayer achieve the best confidence calibration across datasets, often outperforming deep ensembles on ECE.
• Structured dropout yields more diverse ensemble predictions than regular dropout, as shown by lower Interrater Agreement (IA) and sharper reliability diagrams.
• On SVHN, CIFAR-10, and CIFAR-100, structured dropout typically reduces NLL and ECE with comparable or better accuracy than regular dropout.
• MC dropLayer shows robustness to dropout rate choices and can be more calibration-friendly than standard dropout, especially on harder datasets.
• In Bayesian active learning, structured dropout improves accuracy gains from labeled data compared to MC dropout, across acquisition strategies.