[Paper Review] Bayesian Prediction for Artificial Intelligence

Matthew W. Self, Peter Cheeseman|arXiv (Cornell University)|Mar 27, 2013

Bayesian Modeling and Causal Inference7 references3 citations

TL;DR

This paper argues that Bayesian prediction in AI should use model averaging (transduction) rather than selecting a single best model (abduction), especially with small datasets. It demonstrates that transduction avoids misleading predictions and provides closed-form optimal solutions for common model classes.

ABSTRACT

This paper shows that the common method used for making predictions under uncertainty in A1 and science is in error. This method is to use currently available data to select the best model from a given class of models-this process is called abduction-and then to use this model to make predictions about future data. The correct method requires averaging over all the models to make a prediction-we call this method transduction. Using transduction, an AI system will not give misleading results when basing predictions on small amounts of data, when no model is clearly best. For common classes of models we show that the optimal solution can be given in closed form.

Motivation & Objective

• To identify the flaw in standard AI prediction methods that rely on selecting a single best model from a class.
• To address the issue of misleading predictions when no model clearly fits small datasets.
• To propose transduction—averaging predictions across all models—as the correct Bayesian approach.
• To derive closed-form optimal solutions for common model classes under the transduction framework.
• To establish a theoretically sound and practical alternative to model abduction in uncertain prediction tasks.

Proposed method

• Replace model abduction (selecting the best model) with transduction (averaging predictions over all models).
• Apply Bayesian inference to compute model posterior probabilities for all models in a given class.
• Use model averaging to compute predictive distributions, integrating over all models weighted by their posterior probabilities.
• Derive closed-form solutions for prediction under common model classes such as exponential families.
• Formulate the prediction as a marginalization over model space, ensuring robustness to model uncertainty.
• Demonstrate that transduction avoids overconfidence in poorly supported models, especially with limited data.

Experimental results

Research questions

• RQ1 Why does model selection (abduction) lead to misleading predictions in small-data regimes?
• RQ2 How can Bayesian prediction be made robust when no single model is clearly optimal?
• RQ3 What is the theoretically correct method for Bayesian prediction under model uncertainty?
• RQ4 Can closed-form solutions be derived for optimal prediction across common model classes?
• RQ5 How does transduction compare to model selection in terms of predictive accuracy and reliability?

Key findings

• Model selection (abduction) is fundamentally flawed for prediction under uncertainty, especially with small datasets.
• Transduction—averaging over all models—produces more reliable and robust predictions than selecting a single best model.
• For common model classes such as exponential families, the optimal prediction under transduction can be computed in closed form.
• Transduction prevents overconfidence in poorly supported models, reducing the risk of misleading predictions.
• The method ensures consistency with Bayesian principles by fully accounting for model uncertainty in predictions.
• The approach is formally validated and presented as the correct alternative to model abduction in AI prediction tasks.