[论文解读] Bayesian Inference for Large Scale Image Classification

Jonathan Heek, Nal Kalchbrenner|arXiv (Cornell University)|Aug 9, 2019

Adversarial Robustness in Machine Learning参考文献 26被引用 16

一句话总结

本文提出ATMC，一种自适应噪声马尔可夫链蒙特卡洛（MCMC）采样器，通过在训练过程中动态调整动量和噪声，实现了大规模图像分类中的贝叶斯推断。ATMC在ImageNet和CIFAR-10上均优于优化基线方法，无需批量归一化即可实现更高的准确率和更优的不确定性校准，通过后验预测采样实现。

ABSTRACT

Bayesian inference promises to ground and improve the performance of deep neural networks. It promises to be robust to overfitting, to simplify the training procedure and the space of hyperparameters, and to provide a calibrated measure of uncertainty that can enhance decision making, agent exploration and prediction fairness. Markov Chain Monte Carlo (MCMC) methods enable Bayesian inference by generating samples from the posterior distribution over model parameters. Despite the theoretical advantages of Bayesian inference and the similarity between MCMC and optimization methods, the performance of sampling methods has so far lagged behind optimization methods for large scale deep learning tasks. We aim to fill this gap and introduce ATMC, an adaptive noise MCMC algorithm that estimates and is able to sample from the posterior of a neural network. ATMC dynamically adjusts the amount of momentum and noise applied to each parameter update in order to compensate for the use of stochastic gradients. We use a ResNet architecture without batch normalization to test ATMC on the Cifar10 benchmark and the large scale ImageNet benchmark and show that, despite the absence of batch normalization, ATMC outperforms a strong optimization baseline in terms of both classification accuracy and test log-likelihood. We show that ATMC is intrinsically robust to overfitting on the training data and that ATMC provides a better calibrated measure of uncertainty compared to the optimization baseline.

研究动机与目标

• 通过在大规模图像分类任务（如ImageNet）上实现MCMC采样，弥合贝叶斯深度学习中的可扩展性差距。
• 克服MCMC方法与随机正则化技术（如批量归一化和Dropout）之间的不兼容性。
• 开发一种鲁棒的采样算法，降低超参数敏感性，并内在地减少过拟合风险。
• 证明ATMC的后验预测估计在准确率和对数似然性方面均优于标准优化基线。

提出的方法

• 提出ATMC，一种自适应恒温器蒙特卡洛采样器，通过基于参数方差的反馈机制，动态调整每个参数的动量和噪声。
• 引入一种改进的二阶数值积分格式，以提升基于SDE的采样过程的稳定性和收敛性。
• 通过用SELUs、Fixup初始化和权重归一化替代批量归一化，设计ResNet++，实现在无需随机批量统计量情况下的训练稳定性。
• 使用带有自适应噪声注入的随机梯度，通过连续时间SDE近似模型权重的后验分布。
• 通过在多个MCMC样本上平均预测结果，应用后验预测分布以提升不确定性估计。
• 采用学习率退火调度和周期性训练策略，以增强大规模设置下的收敛性和稳定性。

实验结果

研究问题

• RQ1 MCMC采样能否在不依赖批量归一化的情况下，有效扩展到ImageNet等大规模图像分类基准？
• RQ2 ATMC中自适应噪声和动量控制是否相比标准SG-MCMC方法能带来更好的收敛性和鲁棒性？
• RQ3 ATMC的后验预测估计是否在准确率和测试对数似然性方面均优于标准优化基线？
• RQ4 ATMC的不确定性校准与基于优化的模型相比如何，尤其是在低置信度预测区域？

主要发现

• 在CIFAR-10上，ATMC优于强优化基线，使用ResNet++时达到95.5%的测试准确率和0.505的测试对数似然性。
• 在ImageNet上，ATMC的单个后验样本实现74.2%的top-1准确率和1.08的测试对数似然性，优于无BatchNorm的优化基线。
• ATMC的后验预测估计在ImageNet上实现77.5%的top-1准确率和0.883的测试对数似然性，超过使用BatchNorm的优化基线（76.2%准确率，0.947对数似然性）。
• ATMC对过拟合具有内在鲁棒性，无需早停、学习率衰减或随机正则化。
• ATMC的不确定性估计显著优于优化基线，尤其在低置信度预测中表现更优。
• 尽管训练时间长达10倍，ATMC的后验预测在240个周期后达到与BatchNorm优化基线（76.2%准确率）相当的准确率。