[Paper Review] Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory

Sumio Watanabe|arXiv (Cornell University)|Apr 14, 2010

Machine Learning and AlgorithmsComputer Science46 references2,337 citations

TL;DR

In singular learning models, Bayes cross-validation loss and WAIC are asymptotically equivalent as random variables, and their sum with Bayes generalization error is governed by the real log canonical threshold and singular structure.

ABSTRACT

In regular statistical models, the leave-one-out cross-validation is asymptotically equivalent to the Akaike information criterion. However, since many learning machines are singular statistical models, the asymptotic behavior of the cross-validation remains unknown. In previous studies, we established the singular learning theory and proposed a widely applicable information criterion, the expectation value of which is asymptotically equal to the average Bayes generalization loss. In the present paper, we theoretically compare the Bayes cross-validation loss and the widely applicable information criterion and prove two theorems. First, the Bayes cross-validation loss is asymptotically equivalent to the widely applicable information criterion as a random variable. Therefore, model selection and hyperparameter optimization using these two values are asymptotically equivalent. Second, the sum of the Bayes generalization error and the Bayes cross-validation error is asymptotically equal to $2λ/n$, where $λ$ is the real log canonical threshold and $n$ is the number of training samples. Therefore the relation between the cross-validation error and the generalization error is determined by the algebraic geometrical structure of a learning machine. We also clarify that the deviance information criteria are different from the Bayes cross-validation and the widely applicable information criterion.

Motivation & Objective

• Motivate the study by addressing the inadequacy of AIC/BIC in singular models and need for reliable generalization error estimation.
• Define Bayes cross-validation and WAIC within singular learning theory and establish their asymptotic behavior.
• Characterize the relationship between cross-validation, WAIC, and Bayes generalization error through algebraic-geometric invariants.
• Investigate how real log canonical threshold and singular fluctuation determine the asymptotics of generalization and CV errors.

Proposed method

• Define Bayes learning framework with priors, posteriors, and predictive distributions.
• Introduce cross-validation loss CVL(n) and its leave-one-out construction with posterior expectations.
• Use functional cumulants and generating functions to express CVL(n) and WAIC(n) in terms of Yk(n) (k=1..4).
• Prove Theorems 1 and 2: (i) CVL(n) and WAIC(n) share an identical expansion up to Op(1/n^2); (ii) the sum B g(n)+Cv(n) converges to 2λ/(β n) plus terms involving the singular fluctuation ν.
• Relate results to real log canonical threshold λ and birational invariants of the model.

Experimental results

Research questions

• RQ1 Are Bayes cross-validation loss and WAIC asymptotically equivalent as random variables in singular learning models?
• RQ2 How do Bayes generalization error, cross-validation error, and WAIC relate through the real log canonical threshold λ and singular fluctuation ν?
• RQ3 What role do the algebraic-geometric structure of the model (λ, ν) play in the asymptotic behavior of these criteria?

Key findings

• Bayes cross-validation loss and WAIC are asymptotically equivalent as random variables (C v L(n) = WAIC(n) + Op(n^(-3/2)); for β=1, Op(n^(-2)).
• Both CVL(n) and WAIC(n) have expansions dominated by the same functional cumulants Y1(n), Y2(n), and Y3(n).
• The sum of Bayes generalization error and cross-validation error satisfies B g(n)+C v(n) = (β−1)V(n)/n + 2λ/(β n) + o p(1/n); for β=1 this simplifies to 2λ/n + o p(1/n).
• The real log canonical threshold λ and singular fluctuation ν are birational invariants that govern asymptotic behavior, linking CV/WAIC to the algebraic structure of the model.
• Corollary 1 states C v L(n) = WAIC(n) + Op(n^(-3/2)); with β=1, Op(n^(-2)).
• The paper clarifies that deviance information criteria differ from Bayes CV and WAIC in this singular setting.