[Paper Review] Agnostic Learning of Disjunctions on Symmetric Distributions

Vitaly Feldman, Pravesh K. Kothari|arXiv (Cornell University)|May 27, 2014

Machine Learning and Algorithms25 references3 citations

TL;DR

This paper presents an agnostic learning algorithm for disjunctions on symmetric distributions over {0,1}^n, achieving a runtime of n^{O(log(1/ε))} via ℓ₁-approximation using a structured set of basis functions. It proves that low-degree polynomials cannot achieve this approximation on all symmetric distributions, establishing a Ω(√n) degree lower bound for 1/3-approximation, and provides a new, simpler proof for product distributions using degree O(log(1/ε)) polynomials.

ABSTRACT

We consider the problem of approximating and learning disjunctions (or equivalently, conjunctions) on symmetric distributions over $\{0,1\}^n$. Symmetric distributions are distributions whose PDF is invariant under any permutation of the variables. We give a simple proof that for every symmetric distribution $\mathcal{D}$, there exists a set of $n^{O(\log{(1/ε)})}$ functions $\mathcal{S}$, such that for every disjunction $c$, there is function $p$, expressible as a linear combination of functions in $\mathcal{S}$, such that $p$ $ε$-approximates $c$ in $\ell_1$ distance on $\mathcal{D}$ or $\mathbf{E}_{x \sim \mathcal{D}}[ |c(x)-p(x)|] \leq ε$. This directly gives an agnostic learning algorithm for disjunctions on symmetric distributions that runs in time $n^{O( \log{(1/ε)})}$. The best known previous bound is $n^{O(1/ε^4)}$ and follows from approximation of the more general class of halfspaces (Wimmer, 2010). We also show that there exists a symmetric distribution $\mathcal{D}$, such that the minimum degree of a polynomial that $1/3$-approximates the disjunction of all $n$ variables is $\ell_1$ distance on $\mathcal{D}$ is $Ω( \sqrt{n})$. Therefore the learning result above cannot be achieved via $\ell_1$-regression with a polynomial basis used in most other agnostic learning algorithms. Our technique also gives a simple proof that for any product distribution $\mathcal{D}$ and every disjunction $c$, there exists a polynomial $p$ of degree $O(\log{(1/ε)})$ such that $p$ $ε$-approximates $c$ in $\ell_1$ distance on $\mathcal{D}$. This was first proved by Blais et al. (2008) via a more involved argument.

Motivation & Objective

• To develop an efficient agnostic learning algorithm for disjunctions under symmetric distributions, a class that generalizes the uniform distribution.
• To overcome limitations of low-degree polynomial approximation, which fails for certain symmetric distributions.
• To establish tight bounds on the degree of polynomial approximation required for disjunctions under symmetric distributions.
• To provide a simpler proof for the known result that O(log(1/ε))-degree polynomials suffice for ℓ₁-approximation on product distributions.
• To connect the learnability of disjunctions to the hardness of learning sparse parities with noise, showing that faster algorithms would imply breakthroughs in that problem.

Proposed method

• The method relies on constructing a set S of n^{O(log(1/ε))} functions such that any disjunction can be ℓ₁-approximated by a linear combination of functions in S on any symmetric distribution D.
• It uses duality in linear programming to analyze the minimal ℓ₁-error achievable by degree-r polynomials, showing that a dual solution corresponds to a distribution where the error of the best-fitting polynomial matches the optimal approximation error.
• For product distributions, the method leverages noise sensitivity analysis and bounds on the probability of high-weight inputs to show that O(log(1/ε))-degree polynomials suffice for ε-approximation.
• The proof of the Ω(√n) lower bound constructs a specific symmetric distribution D where the disjunction of all n variables cannot be approximated within 1/3 error by any polynomial of degree o(√n).
• It applies the ℓ₁-regression framework from Kalai et al. (2008), using the constructed basis to enable efficient agnostic learning via convex optimization.
• The reduction to learning sparse parities with noise uses correlation bounds between disjunctions and parities, showing that fast learning of disjunctions implies fast learning of k-sparse parities.

Experimental results

Research questions

• RQ1 Can disjunctions be agnostically learned in sub-exponential time on symmetric distributions, and if so, what is the optimal runtime complexity?
• RQ2 Why do standard low-degree polynomial bases fail to approximate disjunctions under all symmetric distributions, and what is the minimal degree required for constant-error approximation?
• RQ3 Can the known result for product distributions—that O(log(1/ε))-degree polynomials ε-approximate any disjunction in ℓ₁—be proven more simply using this framework?
• RQ4 Is there a distribution-specific lower bound on the degree of polynomial approximation for disjunctions on symmetric distributions?
• RQ5 Does the existence of a faster agnostic learning algorithm for disjunctions on symmetric distributions imply faster algorithms for the hard problem of learning sparse parities with noise?

Key findings

• The paper establishes that agnostic learning of disjunctions on any symmetric distribution can be achieved in time n^{O(log(1/ε))}, improving upon the previous n^{O(1/ε⁴)} bound from Wimmer (2010).
• It proves a Ω(√n) lower bound on the degree of any polynomial that 1/3-approximates the full disjunction x₁∨⋯∨xₙ in ℓ₁ distance on a specific symmetric distribution, showing that low-degree polynomials are insufficient in general.
• For product distributions, the paper gives a new, simpler proof that O(log(1/ε))-degree polynomials suffice to ε-approximate any disjunction in ℓ₁ distance, matching the result of Blais et al. (2008) but with a more direct argument.
• The ℓ₁-approximation via a structured basis of size n^{O(log(1/ε))} directly yields an agnostic learning algorithm via ℓ₁-regression, achieving excess error at most ε.
• The paper shows that any algorithm learning disjunctions in time n^{o(log(1/ε))} on the uniform distribution would imply a sub-exponential algorithm for learning k-sparse parities with noise, linking the two problems.
• The reduction demonstrates that improving the runtime of the algorithm beyond n^{O(log(1/ε))} would yield a faster algorithm for a well-known hard problem in learning theory.