[Paper Review] A learning problem that is independent of the set theory ZFC axioms

Shai Ben-David, Pavel Hrubeš+3 more|arXiv (Cornell University)|Nov 14, 2017

Machine Learning and Algorithms16 references3 citations

TL;DR

This paper demonstrates that the learnability of a specific statistical learning problem—Expectation Maximization (EMX) for finite subsets of the real line under countably supported distributions—is independent of the Zermelo-Fraenkel set theory with the Axiom of Choice (ZFC). The key result shows that whether such a class is EMX-learnable depends on the cardinality of the continuum, a value that cannot be determined within ZFC, implying no finite-character combinatorial parameter (like VC-dimension) can fully characterize EMX learnability.

ABSTRACT

We consider the following statistical estimation problem: given a family F of real valued functions over some domain X and an i.i.d. sample drawn from an unknown distribution P over X, find h in F such that the expectation of h w.r.t. P is probably approximately equal to the supremum over expectations on members of F. This Expectation Maximization (EMX) problem captures many well studied learning problems; in fact, it is equivalent to Vapnik's general setting of learning. Surprisingly, we show that the EMX learnability, as well as the learning rates of some basic class F, depend on the cardinality of the continuum and is therefore independent of the set theory ZFC axioms (that are widely accepted as a formalization of the notion of a mathematical proof). We focus on the case where the functions in F are Boolean, which generalizes classification problems. We study the interaction between the statistical sample complexity of F and its combinatorial structure. We introduce a new version of sample compression schemes and show that it characterizes EMX learnability for a wide family of classes. However, we show that for the class of finite subsets of the real line, the existence of such compression schemes is independent of set theory. We conclude that the learnability of that class with respect to the family of probability distributions of countable support is independent of the set theory ZFC axioms. We also explore the existence of a "VC-dimension-like" parameter that captures learnability in this setting. Our results imply that that there exist no "finitary" combinatorial parameter that characterizes EMX learnability in a way similar to the VC-dimension based characterization of binary valued classification problems.

Motivation & Objective

• To investigate whether EMX learnability—finding a function with maximal expectation under an unknown distribution—can be characterized by finite combinatorial parameters.
• To explore the dependence of sample complexity and learnability on set-theoretic assumptions, particularly the cardinality of the continuum.
• To determine whether a VC-dimension-like parameter exists that characterizes EMX learnability in the same way VC-dimension does for binary classification.
• To establish the existence or non-existence of monotone compression schemes for EMX-learnable classes.
• To show that EMX learnability is not robust under different models of set theory, even for weak learnability with constant error.

Proposed method

• Introduce a new variant of sample compression schemes called 'monotone compression schemes' to characterize EMX learnability for union-bounded function classes.
• Prove that monotone compression schemes imply EMX learnability and, under closure conditions, that EMX learnability implies the existence of such schemes.
• Use forcing techniques from set theory to construct two models of ZFC: one where the continuum hypothesis holds ($2^{eth_0} = \aleph_1$) and another where $2^{eth_0} > \aleph_\omega$.
• Show that in the first model, the class of finite subsets of $\mathbb{R}$ is EMX-learnable with constant sample size, while in the second model, no finite sample size suffices.
• Leverage the invariance of finite-character properties across models to show that no such property can capture EMX learnability.
• Analyze the finite superset reconstruction game as a combinatorial tool to relate set-theoretic cardinalities to learnability.

Experimental results

Research questions

• RQ1 Can EMX learnability be characterized by a finite combinatorial parameter analogous to VC-dimension?
• RQ2 Does the existence of a sample compression scheme for EMX learning depend on set-theoretic assumptions about the continuum?
• RQ3 Is EMX learnability of the class of finite subsets of $\mathbb{R}$ invariant across different models of ZFC?
• RQ4 Can weak EMX learnability (e.g., with constant error $1/3$) be captured by a finite-character property?
• RQ5 Is there a set-theoretic independence result for EMX learnability even in the absence of strong closure assumptions on the function class?

Key findings

• The EMX learnability of the class of characteristic functions of finite subsets of $\mathbb{R}$ is independent of ZFC, as it depends on the cardinality of the continuum.
• In a model where $2^{eth_0} = \aleph_1$, this class is EMX-learnable with a constant number of samples.
• In a model where $2^{eth_0} > \aleph_\omega$, no finite number of samples suffices for EMX learning of this class.
• Monotone compression schemes characterize EMX learnability for union-bounded classes, but their existence is independent of ZFC for this specific class.
• No finite-character combinatorial property can capture EMX learnability in the same way VC-dimension captures binary classification learnability.
• The result holds even for weak learnability with constant error $1/3$, indicating a fundamental set-theoretic dependency in the core notion of learnability.