[论文解读] A learning problem that is independent of the set theory ZFC axioms

Shai Ben-David, Pavel Hrubeš+3 more|arXiv (Cornell University)|Nov 14, 2017

Machine Learning and Algorithms参考文献 16被引用 3

一句话总结

本文证明了在可数支撑分布下，实直线有限子集的期望最大化（EMX）统计学习问题的可学习性，独立于策梅洛-弗兰克尔集合论（ZFC）。关键结果表明，此类类别的可学习性取决于连续统的势，而该值在ZFC内无法确定，意味着不存在任何有限组合参数（如VC维）能完全刻画EMX可学习性。

ABSTRACT

We consider the following statistical estimation problem: given a family F of real valued functions over some domain X and an i.i.d. sample drawn from an unknown distribution P over X, find h in F such that the expectation of h w.r.t. P is probably approximately equal to the supremum over expectations on members of F. This Expectation Maximization (EMX) problem captures many well studied learning problems; in fact, it is equivalent to Vapnik's general setting of learning. Surprisingly, we show that the EMX learnability, as well as the learning rates of some basic class F, depend on the cardinality of the continuum and is therefore independent of the set theory ZFC axioms (that are widely accepted as a formalization of the notion of a mathematical proof). We focus on the case where the functions in F are Boolean, which generalizes classification problems. We study the interaction between the statistical sample complexity of F and its combinatorial structure. We introduce a new version of sample compression schemes and show that it characterizes EMX learnability for a wide family of classes. However, we show that for the class of finite subsets of the real line, the existence of such compression schemes is independent of set theory. We conclude that the learnability of that class with respect to the family of probability distributions of countable support is independent of the set theory ZFC axioms. We also explore the existence of a "VC-dimension-like" parameter that captures learnability in this setting. Our results imply that that there exist no "finitary" combinatorial parameter that characterizes EMX learnability in a way similar to the VC-dimension based characterization of binary valued classification problems.

研究动机与目标

• 研究EMX可学习性——即在未知分布下寻找期望最大的函数——是否能通过有限组合参数刻画。
• 探讨样本复杂度与可学习性对集合论假设（特别是连续统势）的依赖性。
• 确定是否存在类似VC维的参数，能以与VC维刻画二值分类可学习性相同的方式刻画EMX可学习性。
• 建立EMX可学习类是否存在单调压缩方案。
• 证明即使在常数误差下的弱可学习性中，EMX可学习性也并非在不同集合论模型下保持不变。

提出的方法

• 引入一种新的样本压缩方案变体，称为‘单调压缩方案’，以刻画并联闭函数类的EMX可学习性。
• 证明单调压缩方案蕴含EMX可学习性，且在闭包条件下，EMX可学习性也蕴含此类方案的存在性。
• 利用集合论中的力迫技术构造两个ZFC模型：一个满足连续统假设（$2^{eth_0} = \aleph_1$），另一个满足$2^{eth_0} > \aleph_\omega$。
• 证明在第一个模型中，$\mathbb{R}$ 的有限子集类可用常数样本大小实现EMX可学习；而在第二个模型中，任何有限样本大小均不足。
• 利用有限特征性质在模型间不变的特性，证明任何此类性质均无法捕捉EMX可学习性。
• 将有限上闭集重构游戏作为组合工具，将集合论中的势与可学习性联系起来。

实验结果

研究问题

• RQ1 EMX可学习性能否通过类似于VC维的有限组合参数刻画？
• RQ2 EMX学习的样本压缩方案是否存在性是否依赖于关于连续统的集合论假设？
• RQ3 实直线有限子集类的EMX可学习性在不同ZFC模型中是否保持不变？
• RQ4 弱EMX可学习性（如常数误差$1/3$）能否由有限特征性质刻画？
• RQ5 即使在函数类缺乏强闭包假设的情况下，EMX可学习性是否仍存在集合论独立性结果？

主要发现

• 实直线有限子集的特征函数类的EMX可学习性独立于ZFC，因其依赖于连续统的势。
• 在$2^{eth_0} = \aleph_1$的模型中，该类可用常数样本数实现EMX可学习。
• 在$2^{eth_0} > \aleph_\omega$的模型中，任何有限样本数均不足以实现该类的EMX学习。
• 单调压缩方案刻画了并联闭类的EMX可学习性，但该类方案的存在性对该特定类而言独立于ZFC。
• 不存在任何有限特征组合性质，能以与VC维刻画二值分类可学习性相同的方式刻画EMX可学习性。
• 即使在常数误差$1/3$的弱可学习性下，该结果依然成立，表明可学习性核心概念存在根本性的集合论依赖性。