[Paper Review] Efficient, Noise-Tolerant, and Private Learning via Boosting

Mark Bun, Marco Carmosino+1 more|arXiv (Cornell University)|Feb 4, 2020

Privacy-Preserving Technologies in Data19 references4 citations

TL;DR

This paper introduces a modular framework for constructing differentially private, noise-tolerant, and efficient boosting algorithms that convert weak learners into strong learners for large-margin halfspaces. By leveraging smooth boosting with privacy-preserving mechanisms—particularly lazy Bregman projections and Gaussian noise—its main result achieves sample complexity independent of dimension, with privacy and accuracy bounds that scale favorably with noise tolerance and confidence.

ABSTRACT

We introduce a simple framework for designing private boosting algorithms. We give natural conditions under which these algorithms are differentially private, efficient, and noise-tolerant PAC learners. To demonstrate our framework, we use it to construct noise-tolerant and private PAC learners for large-margin halfspaces whose sample complexity does not depend on the dimension. We give two sample complexity bounds for our large-margin halfspace learner. One bound is based only on differential privacy, and uses this guarantee as an asset for ensuring generalization. This first bound illustrates a general methodology for obtaining PAC learners from privacy, which may be of independent interest. The second bound uses standard techniques from the theory of large-margin classification (the fat-shattering dimension) to match the best known sample complexity for differentially private learning of large-margin halfspaces, while additionally tolerating random label noise.

Motivation & Objective

• To design a general, modular framework for private boosting that simplifies prior constructions while maintaining privacy and noise tolerance.
• To enable efficient, differentially private PAC learning of large-margin halfspaces with sample complexity independent of input dimension.
• To achieve noise tolerance in the presence of random classification noise by combining privacy guarantees with generalization via fat-shattering dimension.
• To demonstrate that differential privacy can be leveraged as a tool for generalization, not just privacy, through a new privacy-only sample complexity bound.
• To unify smooth boosting, differential privacy, and noise tolerance using a clean theoretical framework based on lazy Bregman projections and measure production.

Proposed method

• Propose a generic boosting schema that ensures privacy and noise tolerance by maintaining smooth distributions over training examples using lazy Bregman projections.
• Use a private weak learner (with Gaussian noise) that maintains a constant advantage over random guessing, even under label noise.
• Apply a private boosting algorithm that combines weak hypotheses via weighted majority, with privacy preserved through zCDP (zero-concentrated differential privacy) and tight composition bounds.
• Introduce a novel privacy-only sample complexity bound that treats privacy as a generalization enabler, independent of standard learning theory.
• Leverage fat-shattering dimension to bound generalization error and ensure low test error with high probability, even under random noise.
• Use Chernoff bounds and concentration inequalities to control failure probabilities from corrupted samples and noise-induced utility loss.

Experimental results

Research questions

• RQ1 Can a unified framework be designed to simultaneously achieve differential privacy, noise tolerance, and efficiency in boosting?
• RQ2 How can differential privacy be used not only to protect data but also to enhance generalization in learning algorithms?
• RQ3 What is the optimal sample complexity for differentially private learning of large-margin halfspaces under random classification noise?
• RQ4 Can privacy-preserving mechanisms like Gaussian noise and Bregman projections be used to maintain smoothness and robustness in boosting?
• RQ5 How does the interplay between privacy, smoothness, and fat-shattering dimension affect the final generalization error in private learning?

Key findings

• The proposed HS-StL algorithm achieves $( ho, au)$-margin halfspace learning with sample complexity $n = O\left(\frac{\sqrt{\log(1/\kappa)\log(1/\delta)\log(\log(1/\kappa)/\beta\tau)}}{\epsilon\kappa\tau^{2}}\right)$, independent of input dimension.
• The privacy-only sample complexity bound demonstrates that differential privacy alone can ensure generalization, with a bound of $O\left(\frac{\sqrt{\log(1/\kappa)\log(1/\delta)\log(\log(1/\kappa)/\beta\tau^{2})}}{\epsilon\alpha\tau^{2}}\right)$.
• The algorithm tolerates random classification noise at rate $O(\alpha\tau)$, with high probability of achieving test error $\leq \alpha$.
• With $T = \frac{1024\log(1/\kappa)}{\tau^{2}}$ rounds, the final hypothesis achieves margin $\gamma = \tau/8$ on all but a $\kappa$ fraction of uncorrupted examples.
• The framework ensures $(\epsilon,\delta)$-differential privacy via $\rho_T$-zCDP and tight composition, with $\rho_T = O\left(\frac{\log(1/\kappa)}{(\kappa n\sigma\tau)^2}\right)$.
• Failure probabilities due to corrupted samples, noise-induced utility loss, or generalization failure are all bounded by $\beta/3$, ensuring overall success probability $1 - \beta$.