[Paper Review] Closure Properties for Private Classification and Online Prediction

Noga Alon, Amos Beimel+2 more|arXiv (Cornell University)|Mar 10, 2020

Machine Learning and Algorithms39 references4 citations

TL;DR

This paper establishes closure properties for differentially private PAC learning and online learning by showing that the Littlestone dimension of a composed hypothesis class $\mathcal{H}'$, formed by aggregating functions from a base class $\mathcal{H}$ via an arbitrary rule $G$, is bounded in terms of the Littlestone dimension of $\mathcal{H}$. It further presents a differentially private algorithm that transforms a private learner for $\mathcal{H}$ into a private learner for $\mathcal{H}'$, achieving near-optimal sample complexity with improved dependence on the original class's complexity.

ABSTRACT

Let~$\cH$ be a class of boolean functions and consider a {\it composed class} $\cH'$ that is derived from~$\cH$ using some arbitrary aggregation rule (for example, $\cH'$ may be the class of all 3-wise majority-votes of functions in $\cH$). We upper bound the Littlestone dimension of~$\cH'$ in terms of that of~$\cH$. As a corollary, we derive closure properties for online learning and private PAC learning. The derived bounds on the Littlestone dimension exhibit an undesirable exponential dependence. For private learning, we prove close to optimal bounds that circumvents this suboptimal dependency. The improved bounds on the sample complexity of private learning are derived algorithmically via transforming a private learner for the original class $\cH$ to a private learner for the composed class~$\cH'$. Using the same ideas we show that any ({\em proper or improper}) private algorithm that learns a class of functions $\cH$ in the realizable case (i.e., when the examples are labeled by some function in the class) can be transformed to a private algorithm that learns the class $\cH$ in the agnostic case.

Motivation & Objective

• To investigate whether the Littlestone dimension of a composed hypothesis class $\mathcal{H}'$, formed by aggregating $k$ functions from a base class $\mathcal{H}$ via an arbitrary rule $G$, can be bounded in terms of the Littlestone dimension of $\mathcal{H}$.
• To establish closure properties for differentially private PAC learning, showing that if a class $\mathcal{H}$ is privately learnable, then so is any $G$-aggregation of $k$ functions from $\mathcal{H}$.
• To design an efficient differentially private algorithm that transforms a private learner for $\mathcal{H}$ into a private learner for $\mathcal{H}'$, with sample complexity close to optimal.
• To bridge the gap between online learning and private learning by leveraging shared combinatorial characterizations, particularly the Littlestone dimension.

Proposed method

• Proposes a transformation algorithm $\mathcal{A}_{\rm ClosureLearn}$ that constructs a private learner for $\mathcal{H}'$ by recursively learning individual components $h_i$ from $\mathcal{H}$ using private empirical learners.
• Uses a union bound and concentration inequalities to ensure that the error of the composed hypothesis $G(h_1,\dots,h_k)$ is bounded with high probability.
• Applies the Sauer-Shelah-Perles Lemma to bound the VC dimension of the composed class $G(\mathcal{H}_1,\dots,\mathcal{H}_k)$ in terms of the sum of the VC dimensions of the individual classes.
• Employs private empirical learning with parameters $\left(\frac{\alpha}{k}, \frac{\beta}{k}\right)$ at each step to ensure differential privacy and accuracy.
• Uses sub-sampling and private boosting techniques to achieve $\varepsilon$-differential privacy and improve accuracy, increasing sample complexity by a factor of $O(1/\varepsilon)$ and $O(1/\alpha)$ respectively.
• Leverages the connection between Littlestone dimension and both online and private learning to derive bounds on regret and sample complexity.

Experimental results

Research questions

• RQ1 Can the Littlestone dimension of a composed class $\mathcal{H}' = \{G(h_1,\dots,h_k) \mid h_i \in \mathcal{H}\}$ be bounded in terms of the Littlestone dimension of the base class $\mathcal{H}$?
• RQ2 Does the existence of a private learner for $\mathcal{H}$ imply the existence of a private learner for $\mathcal{H}'$ with near-optimal sample complexity?
• RQ3 Can a private learner for $\mathcal{H}$ in the realizable case be transformed into a private learner for $\mathcal{H}$ in the agnostic case?
• RQ4 What is the sample complexity of privately learning $\mathcal{H}'$ in terms of the sample complexity of learning $\mathcal{H}$?

Key findings

• The Littlestone dimension of $\mathcal{H}'$, the class of $k$-wise aggregations of functions from $\mathcal{H}$, is bounded by a function of the Littlestone dimension of $\mathcal{H}$, resolving the closure property for online learning.
• The proposed algorithm $\mathcal{A}_{\rm ClosureLearn}$ achieves $O(\alpha)$ error with probability at least $1 - O(\beta + \delta \sum_{i=1}^k |S_i|)$, ensuring both accuracy and differential privacy.
• The sample complexity of $\mathcal{A}_{\rm ClosureLearn}$ is $\tilde{O}\left(\sum_{i=1}^k m_i\left(\frac{\alpha}{k}, \frac{\beta}{k}, 1, \delta\right)\right) \cdot \mathrm{poly}(k, 1/\alpha, \log(1/\beta))$, with improved dependence on $k$ and $\alpha$ compared to prior bounds.
• The algorithm achieves $\varepsilon$-differential privacy by increasing the sample complexity by a factor of $O(1/\varepsilon)$ via sub-sampling, and can be boosted to any desired accuracy and confidence.
• The method circumvents the exponential dependence on $k$ seen in standard Littlestone dimension bounds by using a recursive, private learning approach with careful error propagation.
• The results show that private learning is closed under aggregation: if $\mathcal{H}$ is privately learnable, so is $\mathcal{H}'$, even in the agnostic setting, via a transformation from the realizable case.