[Paper Review] Sample and Computationally Efficient Learning Algorithms under S-Concave Distributions

Maria-Florina Balcan, Hongyang Zhang|arXiv (Cornell University)|Mar 22, 2017

Machine Learning and Algorithms9 citations

TL;DR

This paper introduces sample- and computation-efficient learning algorithms for linear and intersection-of-halfspaces classifiers under s-concave distributions, a broad class generalizing log-concave and fat-tailed distributions like t- and Pareto. By developing new convex geometry tools, the authors establish bounds on disagreement probability and coefficient, enabling significant generalizations of prior results in margin-based, disagreement-based, and passive active learning under these distributions.

ABSTRACT

We provide new results for noise-tolerant and sample-efficient learning algorithms under $s$-concave distributions. The new class of $s$-concave distributions is a broad and natural generalization of log-concavity, and includes many important additional distributions, e.g., the Pareto distribution and $t$-distribution. This class has been studied in the context of efficient sampling, integration, and optimization, but much remains unknown about the geometry of this class of distributions and their applications in the context of learning. The challenge is that unlike the commonly used distributions in learning (uniform or more generally log-concave distributions), this broader class is not closed under the marginalization operator and many such distributions are fat-tailed. In this work, we introduce new convex geometry tools to study the properties of $s$-concave distributions and use these properties to provide bounds on quantities of interest to learning including the probability of disagreement between two halfspaces, disagreement outside a band, and the disagreement coefficient. We use these results to significantly generalize prior results for margin-based active learning, disagreement-based active learning, and passive learning of intersections of halfspaces. Our analysis of geometric properties of $s$-concave distributions might be of independent interest to optimization more broadly.

Motivation & Objective

• To extend noise-tolerant and sample-efficient learning algorithms beyond log-concave distributions to the broader class of s-concave distributions.
• To address the challenge that s-concave distributions are not closed under marginalization and often have fat tails, complicating prior algorithmic assumptions.
• To develop new convex geometry tools to analyze geometric properties of s-concave distributions relevant to learning theory.
• To generalize existing results in margin-based active learning, disagreement-based active learning, and passive learning of intersections of halfspaces to s-concave settings.
• To establish tight sample and label complexity bounds under s-concave distributions, including lower bounds.

Proposed method

• Introduce a new class of s-concave distributions defined by the concavity of f(x)^s for s < 0, generalizing log-concave distributions.
• Develop novel convex geometry tools to analyze marginalization properties and hyperplane disagreement under s-concave distributions.
• Establish closed-form bounds on the probability of disagreement between two halfspaces and the disagreement coefficient under isotropic s-concave distributions.
• Use these geometric bounds to derive sample complexity and label complexity guarantees for active and passive learning algorithms.
• Design and analyze three new algorithms: margin-based active learning under realizable and noisy settings, and learning intersections of halfspaces under s-concave distributions.
• Leverage concentration inequalities and pseudo-dimension bounds to prove generalization and consistency of the proposed learning procedures.

Experimental results

Research questions

• RQ1 How can learning algorithms be made sample- and computation-efficient under the broader class of s-concave distributions, which include fat-tailed distributions like t- and Pareto?
• RQ2 What are the fundamental geometric properties of s-concave distributions that enable tighter bounds on disagreement probability and coefficient?
• RQ3 To what extent can existing active and passive learning algorithms for halfspaces be generalized to s-concave distributions, and what are the resulting sample and label complexity improvements?
• RQ4 How do the geometric properties of s-concave distributions compare to those of log-concave and nearly-log-concave distributions in terms of algorithmic feasibility and performance?
• RQ5 What are the information-theoretic lower bounds on sample and label complexity for learning homogeneous halfspaces under s-concave distributions?

Key findings

• The marginal distribution of an isotropic s-concave distribution over m variables is isotropic γ(s,m)-concave, establishing a weak closure property under marginalization.
• For any two unit vectors u and v, the disagreement probability Pr[sign(u·x) ≠ sign(v·x)] is lower bounded by f₁(s,n)θ(u,v), where θ(u,v) is the angle between them.
• The disagreement coefficient under s-concave distributions is bounded by f₃(s,n), enabling improved generalization bounds in active learning.
• The sample complexity for passive PAC learning of homogeneous linear separators under s-concave distributions is Ω(n log(f₁(s,n)/ε)), matching known lower bounds up to logarithmic factors.
• The label complexity for active learning under s-concave distributions is Ω(log M_D(C,ε)), showing that improvements are information-theoretically possible under this class.
• The proposed algorithms achieve sample and label complexity bounds that generalize and improve upon prior results for log-concave and nearly-log-concave distributions.