[Paper Review] A Deep Learning Method for Comparing Bayesian Hierarchical Models

Lasse Elsemüller, Martin Schnuerch+2 more|arXiv (Cornell University)|Jan 27, 2023

Bayesian Modeling and Causal Inference4 citations

TL;DR

This paper proposes a deep learning method for Bayesian hierarchical model comparison (BMC) that enables efficient, amortized inference of Bayes factors and posterior model probabilities across complex hierarchical models, including those with implicit likelihoods. By leveraging a hierarchical neural network architecture within the BayesFlow framework, the method achieves high accuracy comparable to bridge sampling while drastically reducing computational cost, with demonstrated success on intractable cognitive models and transfer learning benefits.

ABSTRACT

Bayesian model comparison (BMC) offers a principled approach for assessing the relative merits of competing computational models and propagating uncertainty into model selection decisions. However, BMC is often intractable for the popular class of hierarchical models due to their high-dimensional nested parameter structure. To address this intractability, we propose a deep learning method for performing BMC on any set of hierarchical models which can be instantiated as probabilistic programs. Since our method enables amortized inference, it allows efficient re-estimation of posterior model probabilities and fast performance validation prior to any real-data application. In a series of extensive validation studies, we benchmark the performance of our method against the state-of-the-art bridge sampling method and demonstrate excellent amortized inference across all BMC settings. We then showcase our method by comparing four hierarchical evidence accumulation models that have previously been deemed intractable for BMC due to partly implicit likelihoods. Additionally, we demonstrate how transfer learning can be leveraged to enhance training efficiency. We provide reproducible code for all analyses and an open-source implementation of our method.

Motivation & Objective

• To address the intractability of Bayesian model comparison (BMC) for hierarchical models due to high-dimensional, nested parameter spaces.
• To develop a scalable, simulation-based method for BMC that bypasses explicit marginalization over high-dimensional parameter spaces.
• To enable fast, repeated model comparison prior to real-data analysis through amortized inference.
• To extend BMC applicability to hierarchical models with implicit likelihoods, such as simulation-based models in cognitive science.
• To provide a diagnostic for model calibration and support transfer learning to improve training efficiency.

Proposed method

• The method employs a hierarchical neural network architecture to directly approximate Bayes factors and posterior model probabilities from probabilistic programs.
• It is built upon the BayesFlow framework, enabling end-to-end training on simulated data from competing hierarchical models.
• The model is trained to predict model evidence and posterior model probabilities using simulated data, allowing amortized inference across new datasets.
• A calibration error metric is computed to assess self-consistency and reliability of the neural approximation.
• The approach supports transfer learning by fine-tuning pre-trained networks on related model families, reducing training time for new tasks.
• The method is applicable to both models with explicit likelihoods and those accessible only as Monte Carlo simulators (implicit likelihoods).

(a) Marginal Likelihoods

Experimental results

Research questions

• RQ1 Can a deep learning model accurately approximate Bayes factors and posterior model probabilities for hierarchical models with complex, high-dimensional parameter structures?
• RQ2 How does the performance of the proposed deep learning method compare to state-of-the-art bridge sampling in terms of accuracy and computational efficiency?
• RQ3 Can the method be effectively applied to hierarchical models with implicit likelihoods, such as those used in cognitive psychology?
• RQ4 To what extent can transfer learning reduce training time when applying the method to new but related hierarchical models?
• RQ5 How reliable is the method’s self-calibration error as a diagnostic for model consistency?

Key findings

• The deep learning method achieves performance on par with bridge sampling across all tested BMC settings, demonstrating high accuracy in approximating Bayes factors and posterior model probabilities.
• The method enables amortized inference, allowing rapid re-estimation of model probabilities across new datasets without retraining from scratch.
• The approach successfully compares four hierarchical evidence accumulation models with partly implicit likelihoods, which were previously intractable for BMC.
• The method's calibration error provides a reliable diagnostic for model consistency, with low error indicating well-calibrated posterior estimates.
• Transfer learning significantly reduces training time when adapting the model to new hierarchical models, enhancing scalability and reusability.
• An open-source implementation and reproducible code are provided, supporting full reproducibility and community adoption.

(b) Posterior Model Probabilities