[Paper Review] Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning

Ruqi Zhang, Chunyuan Li+3 more|arXiv (Cornell University)|Feb 11, 2019

Machine Learning and AlgorithmsComputer Science57 references75 citations

TL;DR

This paper introduces cyclical SG-MCMC (cSG-MCMC) with a cyclical stepsize schedule to automatically explore multimodal posterior distributions in Bayesian deep learning, and provides non-asymptotic convergence theory and extensive experiments including ImageNet.

ABSTRACT

The posteriors over neural network weights are high dimensional and multimodal. Each mode typically characterizes a meaningfully different representation of the data. We develop Cyclical Stochastic Gradient MCMC (SG-MCMC) to automatically explore such distributions. In particular, we propose a cyclical stepsize schedule, where larger steps discover new modes, and smaller steps characterize each mode. We also prove non-asymptotic convergence of our proposed algorithm. Moreover, we provide extensive experimental results, including ImageNet, to demonstrate the scalability and effectiveness of cyclical SG-MCMC in learning complex multimodal distributions, especially for fully Bayesian inference with modern deep neural networks.

Motivation & Objective

• Motivate Bayesian deep learning as a principled approach to quantify uncertainty in neural network weights.
• Develop a cyclical stepsize SG-MCMC method to efficiently explore highly multimodal weight posteriors.
• Provide theoretical guarantees for non-asymptotic convergence under cyclical scheduling.
• Demonstrate scalability and effectiveness of cSG-MCMC through large-scale experiments (e.g., ImageNet) and uncertainty estimation.

Proposed method

• Propose a cyclical cosine stepsize schedule for SG-MCMC that alternates between large steps for exploration and small steps for local sampling.
• Introduce a two-stage procedure: exploration (large steps, high perturbation) and sampling (small steps, local density estimation).
• Use a system temperature to control exploration versus sampling, with T=0 for burn-in and T=1 for sampling, and a beta threshold to switch stages within each cycle.
• Treat exploration as warm restarts that periodically reinitialize with large steps to escape current modes.
• Provide a weighted sample combination scheme across cycles to combine information from different modes.

Experimental results

Research questions

• RQ1 Can cyclical SG-MCMC effectively explore and characterize multimodal weight posteriors in modern neural networks?
• RQ2 Does the cyclical schedule improve mixing and uncertainty estimation relative to traditional decreasing-step SG-MCMC?
• RQ3 What are the theoretical (non-asymptotic) convergence guarantees for cSG-MCMC under cyclical stepsizes?
• RQ4 How does cSG-MCMC perform on large-scale tasks (e.g., ImageNet) and in uncertainty quantification tasks?

Key findings

• cSG-MCMC discovers and characterizes multiple modes in multimodal distributions using cycles, outperforming traditional SGLD in mode exploration.
• On CIFAR-10/100 with ResNet-18, cyclical methods yield lower testing errors than traditional SG-MCMC and Snapshot ensembles, with improved diversity.
• On ImageNet with ResNet-50, cSG-MCMC achieves the lowest predictive NLL among tested methods, indicating stronger uncertainty modeling.
• Visualization shows weight-space samples from cSG-MCMC form multiple clusters, indicating exploration of diverse modes.
• Uncertainty evaluation on notMNIST shows cSG-MCMC provides better calibrated predictive uncertainties by exploring more weight-space modes.