[论文解读] Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory

Sumio Watanabe|arXiv (Cornell University)|Apr 14, 2010

Machine Learning and Algorithms参考文献 46被引用 2,337

一句话总结

在单一学习模型中，Bayes cross-validation loss 和 WAIC 作为随机变量在渐近意义上等价，它们与 Bayes generalization error 的和由 real log canonical threshold 和奇异结构所支配。

ABSTRACT

In regular statistical models, the leave-one-out cross-validation is asymptotically equivalent to the Akaike information criterion. However, since many learning machines are singular statistical models, the asymptotic behavior of the cross-validation remains unknown. In previous studies, we established the singular learning theory and proposed a widely applicable information criterion, the expectation value of which is asymptotically equal to the average Bayes generalization loss. In the present paper, we theoretically compare the Bayes cross-validation loss and the widely applicable information criterion and prove two theorems. First, the Bayes cross-validation loss is asymptotically equivalent to the widely applicable information criterion as a random variable. Therefore, model selection and hyperparameter optimization using these two values are asymptotically equivalent. Second, the sum of the Bayes generalization error and the Bayes cross-validation error is asymptotically equal to $2λ/n$, where $λ$ is the real log canonical threshold and $n$ is the number of training samples. Therefore the relation between the cross-validation error and the generalization error is determined by the algebraic geometrical structure of a learning machine. We also clarify that the deviance information criteria are different from the Bayes cross-validation and the widely applicable information criterion.

研究动机与目标

• 通过解决 AIC/BIC 在奇异模型中的不足以及需要可靠泛化误差估计来驱动研究。
• 在奇异学习理论中定义 Bayes cross-validation 和 WAIC，并建立它们的渐近行为。
• 通过代数几何不变量描述交叉验证、WAIC 与 Bayes generalization error 之间的关系。
• 研究 real log canonical threshold 与奇异波动如何决定泛化误差与 CV 误差的渐近性质。

提出的方法

• 在先验、后验和预测分布下定义 Bayes 学习框架。
• 引入 cross-validation loss CVL(n) 及其留一法构造与后验期望。
• 使用泛函累积量和生成函数将 CVL(n) 与 WAIC(n) 表示为关于 Yk(n) (k=1..4) 的形式。
• 证明定理 1 和 2： (i) CVL(n) 与 WAIC(n) 具有相同的展开式，直到 Op(1/n^2)；(ii) 融合项 B g(n)+Cv(n) 收敛到 2λ/(β n) 加上涉及奇异波动 ν 的项。
• 将结果与 real log canonical threshold λ 及模型的有 birational 不变量联系起来。

实验结果

研究问题

• RQ1 在奇异学习模型中，Bayes cross-validation loss 和 WAIC 是否作为随机变量在渐近意义上等价？
• RQ2 Bayes generalization error、cross-validation error 与 WAIC 如何通过 real log canonical threshold λ 与 singular fluctuation ν 相互关系？
• RQ3 模型的代数几何结构（λ, ν）在这些准则的渐近行为中扮演怎样的角色？

主要发现

• Bayes cross-validation loss 与 WAIC 在渐近意义上作为随机变量等价（CvL(n) = WAIC(n) + Op(n^(-3/2))；对 β=1，Op(n^(-2))）。
• CVL(n) 与 WAIC(n) 的展开都由同样的函数累积量 Y1(n)、Y2(n) 与 Y3(n) 主导。
• Bayes generalization error 与 cross-validation error 的和满足 B g(n)+C v(n) = (β−1)V(n)/n + 2λ/(β n) + o p(1/n)；对 β=1 这一式化简为 2λ/n + o p(1/n)。
• real log canonical threshold λ 与奇异波动 ν 是支配渐近行为的有 birational 不变量，将 CV/WAIC 与模型的代数结构联系起来。
• Corollary 1 表示 C v L(n) = WAIC(n) + Op(n^(-3/2)); 当 β=1 时，为 Op(n^(-2))。
• 本文澄清在此奇异设定中 deviance information criteria 与 Bayes CV 与 WAIC 的差别。