[Paper Review] Adaptive variational Bayes: Optimality, computation and applications

Ilsang Ohn, Lizhen Lin|arXiv (Cornell University)|Sep 7, 2021

Model Reduction and Neural Networks107 references4 citations

TL;DR

This paper proposes an adaptive variational Bayes framework that combines variational posteriors from multiple models using optimal weights, achieving adaptive posterior contraction rates under general conditions. The method ensures computational tractability and optimal frequentist frequentist performance even with large model collections, including sparse and deep learning models.

ABSTRACT

In this paper, we explore adaptive inference based on variational Bayes. Although several studies have been conducted to analyze the contraction properties of variational posteriors, there is still a lack of a general and computationally tractable variational Bayes method that performs adaptive inference. To fill this gap, we propose a novel adaptive variational Bayes framework, which can operate on a collection of models. The proposed framework first computes a variational posterior over each individual model separately and then combines them with certain weights to produce a variational posterior over the entire model. It turns out that this combined variational posterior is the closest member to the posterior over the entire model in a predefined family of approximating distributions. We show that the adaptive variational Bayes attains optimal contraction rates adaptively under very general conditions. We also provide a methodology to maintain the tractability and adaptive optimality of the adaptive variational Bayes even in the presence of an enormous number of individual models, such as sparse models. We apply the general results to several examples, including deep learning and sparse factor models, and derive new and adaptive inference results. In addition, we characterize an implicit regularization effect of variational Bayes and show that the adaptive variational posterior can utilize this.

Motivation & Objective

• Address the lack of a general, computationally tractable variational Bayes method that achieves adaptive inference across multiple models.
• Develop a framework that maintains optimal posterior contraction rates without requiring model-specific priors or variational families.
• Ensure computational feasibility when dealing with a large number of models, such as in sparse or high-dimensional settings.
• Demonstrate the method's adaptivity and optimality in complex models like deep neural networks and sparse factor models.
• Characterize the implicit regularization effect of variational Bayes and show how it enhances inference performance.

Proposed method

• Compute a separate variational posterior for each individual model in a collection using standard variational inference.
• Combine these individual variational posteriors using optimal weights to form a global adaptive variational posterior.
• The resulting adaptive variational posterior is the closest approximation to the true posterior within a predefined family of distributions.
• Use a two-stage procedure: first variational inference per model, then model combination via weighted averaging.
• Leverage theoretical results from Zhang & Gao (2020) to ensure contraction rate optimality under mild regularity conditions.
• Apply implicit regularization effects of variational Bayes to improve performance in high-dimensional and sparse models.

Experimental results

Research questions

• RQ1 Can a general, computationally tractable variational Bayes method achieve adaptive posterior contraction rates across diverse models?
• RQ2 How can variational posteriors from multiple models be optimally combined to approximate the true posterior?
• RQ3 Does the proposed framework maintain adaptive optimality when the number of models is large or in high-dimensional settings?
• RQ4 What is the role of implicit regularization in variational Bayes, and how can it be leveraged in the adaptive framework?
• RQ5 Can the method achieve optimal contraction rates in complex models such as deep neural networks and sparse factor models?

Key findings

• The adaptive variational posterior achieves optimal posterior contraction rates under very general conditions, including in nonparametric and high-dimensional settings.
• The method maintains computational tractability even when the number of models is large, such as in sparse model collections.
• The framework outperforms model selection-based variational Bayes by providing a closer approximation to the true posterior.
• The adaptive variational posterior inherits and leverages the implicit regularization effect of variational Bayes, improving estimation stability.
• Theoretical results are validated in deep learning and sparse factor models, yielding new adaptive inference results with optimal rates.
• The method is shown to be optimal in terms of frequentist risk, with contraction rates matching the minimax optimal rates under mild assumptions.