[Paper Review] Privately Learning Thresholds: Closing the Exponential Gap

Haim Kaplan, Katrina Ligett+3 more|arXiv (Cornell University)|Nov 22, 2019

Privacy-Preserving Technologies in Data11 references4 citations

TL;DR

This paper closes the exponential gap between the best-known lower and upper bounds for differentially private learning of threshold functions. It introduces a novel algorithm based on recursive hashing and path construction that achieves a sample complexity of Õ((log* |X|)^1.5), significantly improving prior work and nearly settling the asymptotic sample complexity for this problem under approximate differential privacy.

ABSTRACT

We study the sample complexity of learning threshold functions under the constraint of differential privacy. It is assumed that each labeled example in the training data is the information of one individual and we would like to come up with a generalizing hypothesis $h$ while guaranteeing differential privacy for the individuals. Intuitively, this means that any single labeled example in the training data should not have a significant effect on the choice of the hypothesis. This problem has received much attention recently; unlike the non-private case, where the sample complexity is independent of the domain size and just depends on the desired accuracy and confidence, for private learning the sample complexity must depend on the domain size $X$ (even for approximate differential privacy). Alon et al. (STOC 2019) showed a lower bound of $Ω(\log^*|X|)$ on the sample complexity and Bun et al. (FOCS 2015) presented an approximate-private learner with sample complexity $ ilde{O}\left(2^{\log^*|X|} ight)$. In this work we reduce this gap significantly, almost settling the sample complexity. We first present a new upper bound (algorithm) of $ ilde{O}\left(\left(\log^*|X| ight)^2 ight)$ on the sample complexity and then present an improved version with sample complexity $ ilde{O}\left(\left(\log^*|X| ight)^{1.5} ight)$. Our algorithm is constructed for the related interior point problem, where the goal is to find a point between the largest and smallest input elements. It is based on selecting an input-dependent hash function and using it to embed the database into a domain whose size is reduced logarithmically; this results in a new database, an interior point of which can be used to generate an interior point in the original database in a differentially private manner.

Motivation & Objective

• To close the exponential gap between the best-known lower and upper bounds on sample complexity for differentially private learning of threshold functions.
• To design a private algorithm that achieves near-optimal sample complexity for learning thresholds under approximate differential privacy.
• To develop a new technique for private interior point finding that enables efficient recursive reduction of domain size.
• To improve upon the prior upper bound of Õ(2^{log* |X|}) by reducing it to Õ((log* |X|)^1.5).
• To establish tight utility and privacy guarantees for the proposed algorithm in terms of accuracy, privacy parameters, and domain size.

Proposed method

• Proposes a recursive algorithm called HeavyPaths that combines StoppingPoint, OneRandomPath, and LevelUp to privately identify an interior point in the data.
• Uses input-dependent hash functions to embed the original database into a smaller domain, reducing size logarithmically at each level.
• Employs a depth-based path construction mechanism to ensure that the selected point is an interior point with high probability.
• Applies advanced differential privacy composition theorems to bound privacy loss across multiple recursive steps and subroutines.
• Uses a multi-level reduction strategy where each level maintains a private estimate of the depth of the interior point, enabling recursive refinement.
• Introduces a stopping criterion based on a statistical test to determine the appropriate recursion depth, ensuring utility while preserving privacy.

Experimental results

Research questions

• RQ1 What is the tightest possible sample complexity for differentially private learning of threshold functions under approximate differential privacy?
• RQ2 Can the gap between the Ω(log* |X|) lower bound and the Õ(2^{log* |X|}) upper bound be significantly reduced?
• RQ3 Is it possible to design a private algorithm that achieves sub-exponential dependence on log* |X| while maintaining constant accuracy and confidence?
• RQ4 How can recursive domain reduction via private hashing be used to construct a private interior point finder with strong utility guarantees?
• RQ5 Can the privacy analysis be refined to achieve tighter bounds on ε and δ in terms of log* |X| and n?

Key findings

• The paper achieves a new upper bound of Õ((log* |X|)^1.5) on the sample complexity for differentially private threshold learning, significantly improving over the prior Õ(2^{log* |X|}) bound.
• The algorithm HeavyPaths returns an interior point of the input database with probability at least 1 - O(δ · log* |X|) when n ≥ Ω((log* |X|)/ε) · log(1/(δ · log* |X|)).
• The algorithm is (O(ε·√(log* |X|·log(1/(δ·log* |X|)))) + ε²·log* |X| + ε·log n, O(δ·(n + log* |X|)·e^{O(ε·√(log* |X|·log(1/(δ·log* |X|))))}))-differentially private.
• The method reduces the domain size logarithmically at each recursive level using private hashing, enabling efficient and private path construction.
• The analysis establishes that the algorithm maintains high utility across all levels of recursion, ensuring that the final output is a valid interior point with high probability.
• The results nearly settle the sample complexity of private threshold learning, closing the long-standing exponential gap between known lower and upper bounds.