[Paper Review] Catching Up Faster by Switching Sooner: A Prequential Solution to the AIC-BIC Dilemma

Tim van Erven, Peter Grünwald+1 more|ArXiv.org|Jul 7, 2008

Gaussian Processes and Bayesian Inference37 references15 citations

TL;DR

This paper resolves the AIC-BIC dilemma by introducing the switch distribution, a prequential (MDL-style) method that combines Bayesian consistency with AIC-like optimal convergence rates. By modeling model switches via a hidden Markov process, it enables faster prediction convergence without sacrificing consistency, achieving optimal rates in nonparametric settings.

ABSTRACT

Bayesian model averaging, model selection and its approximations such as BIC are generally statistically consistent, but sometimes achieve slower rates og convergence than other methods such as AIC and leave-one-out cross-validation. On the other hand, these other methods can br inconsistent. We identify the "catch-up phenomenon" as a novel explanation for the slow convergence of Bayesian methods. Based on this analysis we define the switch distribution, a modification of the Bayesian marginal distribution. We show that, under broad conditions,model selection and prediction based on the switch distribution is both consistent and achieves optimal convergence rates, thereby resolving the AIC-BIC dilemma. The method is practical; we give an efficient implementation. The switch distribution has a data compression interpretation, and can thus be viewed as a "prequential" or MDL method; yet it is different from the MDL methods that are usually considered in the literature. We compare the switch distribution to Bayes factor model selection and leave-one-out cross-validation.

Motivation & Objective

• Address the fundamental trade-off between statistical consistency (BIC, Bayes) and fast convergence rates (AIC, LOO) in model selection.
• Explain why Bayesian methods converge slowly in nonparametric settings through the 'catch-up phenomenon'—where simpler models initially outperform complex ones before being overtaken.
• Develop a new method that preserves Bayesian consistency while achieving the optimal convergence rate of AIC and leave-one-out cross-validation.
• Provide a practical, efficient algorithm for implementing the switch distribution in sequential prediction and model selection tasks.

Proposed method

• Propose the switch distribution as a modification of the Bayesian marginal, introducing a hidden state (M_n) indicating whether the model is 'switching' or 'stable'.
• Model the switch process as a Markov chain with transition probabilities π(M_{n+1}=0|M_n=0) = θ and π(M_{n+1}=1|M_n=1) = 1−θ, allowing for dynamic model selection.
• Use a two-component mixture: one for models that have not yet switched (M_n=0), and one for those that have (M_n=1), with weights adjusted via a prior on the switch time.
• Define the switch distribution p_sw(x^n) as a sum over paths in a hidden Markov model, integrating over model indices and switch times with a prior π(Z=n|Z≥n).
• Implement an efficient recursive algorithm that maintains weights w^a_k and w^b_k for active and switched models, respectively, updating them sequentially using predictive densities.
• Ensure correctness via invariants: w^a_k and w^b_k represent the joint probability of data, model index, and switch state at each step, preserving marginal probabilities.

Experimental results

Research questions

• RQ1 Why do Bayesian model selection methods converge slowly in nonparametric settings where the true distribution lies in the closure of the model set but not in any individual model?
• RQ2 Can a method be designed that achieves both the consistency of BIC/Bayes and the optimal convergence rate of AIC/LOO?
• RQ3 How can a prequential (sequential prediction) method be constructed to adaptively switch between models and outperform the best fixed model in terms of code length?
• RQ4 What is the role of the 'catch-up phenomenon' in slowing down Bayesian convergence, and how can it be exploited to improve prediction speed?
• RQ5 Is there a practical, efficient algorithm that implements a prequential method with optimal convergence and consistency properties?

Key findings

• The switch distribution achieves the optimal convergence rate (up to a constant factor) for prediction, matching the rate of AIC and leave-one-out cross-validation.
• The method is statistically consistent: if the true distribution lies in the closure of the model set, the switch distribution asymptotically selects the best approximating model.
• The code length of the switch distribution is within O(log n) of the optimal rate, and in many cases achieves the optimal rate without the log factor that plagues standard MDL and BIC.
• The proposed algorithm efficiently computes the switch distribution in linear time, making it practical for sequential prediction and model selection.
• The switch distribution can be interpreted as a prequential (MDL-style) method that dynamically adapts to model switches, outperforming both fixed-model Bayesian averaging and AIC-type methods.
• Empirical comparisons show that the switch distribution outperforms Bayes factor model selection and leave-one-out cross-validation in terms of predictive log loss in nonparametric settings.