[Paper Review] A framework for probabilistic inferences from imperfect models

Meng Li, David B. Dunson|arXiv (Cornell University)|Nov 4, 2016

Gaussian Processes and Bayesian Inference27 references3 citations

TL;DR

This paper introduces absolute model probabilities, a new framework for evaluating imperfect statistical models by measuring their quality relative to a nonparametric Bayesian reference using Kullback-Leibler divergence. The D-probabilities it produces provide a robust, prior-insensitive alternative to traditional posterior model probabilities, automatically penalizing model complexity and enabling reliable model comparison even when no model is exactly correct.

ABSTRACT

The Bayesian paradigm provides a natural way to deal with uncertainty in model selection through assigning each model in a list of models under consideration a posterior probability. Unfortunately, this framework relies on the assumption that one of the models in the list is the true model. When this assumption is violated and all the models are imperfect, interpretation of posterior model probabilities is unclear. We propose a new concept of {\em absolute} model probabilities, which measure the quality of imperfect models. This concept leads to divergence-based estimates ({\em D-probabilities}) relying on evaluating parametric models relative to a nonparametric Bayesian reference using Kullback-Leibler divergence. While providing good-of-fit assessment, D-probabilities avoid some of the pitfalls of usual posterior model probabilities including large sensitivity to prior choice. In an application to linear model selection against a Gaussian process reference, we provide simple analytic forms for routine implementation and show that D-probabilities automatically penalize model complexity. Some asymptotic properties of this framework are described. Absolute model probabilities have several interesting probabilistic interpretations, and can potentially be applied in broad problems. The framework is illustrated through simulations and applications.

Motivation & Objective

• To address the limitation of Bayesian model selection when all models are imperfect, challenging the assumption that one model in the set is true.
• To develop a coherent framework for assessing model quality independent of model truth, focusing on fit and complexity.
• To provide a robust alternative to posterior model probabilities that avoids large sensitivity to prior distributions.
• To enable practical model comparison in settings where standard Bayesian model averaging fails due to model misspecification.

Proposed method

• Proposes absolute model probabilities as a measure of model quality, defined relative to a nonparametric Bayesian reference distribution.
• Uses Kullback-Leibler divergence to quantify the discrepancy between parametric models and the nonparametric reference.
• Derives D-probabilities as divergence-based estimates that reflect how well a model fits the reference, with lower divergence indicating higher quality.
• Applies the framework to linear models against a Gaussian process reference, yielding analytic expressions for routine use.
• Incorporates automatic penalization of model complexity through the divergence measure, without requiring explicit prior adjustments.
• Establishes asymptotic properties of the D-probability framework under regularity conditions.

Experimental results

Research questions

• RQ1 How can model quality be meaningfully assessed when all models in a set are imperfect?
• RQ2 What is a principled alternative to posterior model probabilities that avoids sensitivity to prior choice?
• RQ3 Can a divergence-based measure provide automatic model complexity penalization without relying on marginal likelihoods?
• RQ4 How do D-probabilities behave asymptotically under model misspecification?
• RQ5 What are the probabilistic interpretations and practical implications of absolute model probabilities?

Key findings

• D-probabilities provide a coherent, prior-insensitive measure of model quality that remains interpretable even when no model is true.
• The framework automatically penalizes model complexity through the Kullback-Leibler divergence, eliminating the need for additional regularization.
• Analytic forms for D-probabilities are derived in the context of linear models and Gaussian process references, enabling straightforward implementation.
• D-probabilities avoid the pathologies of posterior model probabilities, such as extreme sensitivity to prior distributions.
• The framework exhibits desirable asymptotic properties, supporting consistent model ranking under regularity conditions.
• Absolute model probabilities offer a new probabilistic interpretation of model fit, extending Bayesian inference beyond the assumption of model truth.