[Paper Review] Private and polynomial time algorithms for learning Gaussians and beyond

Hassan Ashtiani, Christopher Liaw|arXiv (Cornell University)|Nov 22, 2021

Machine Learning and Algorithms34 references4 citations

TL;DR

This paper presents a general framework for transforming non-private statistical estimation algorithms into differentially private, polynomial-time methods. It achieves optimal sample complexity for private learning of unrestricted Gaussians in R^d, matching the best-known theoretical bounds up to logarithmic factors, and provides the first efficient private robust learning algorithm with sample complexity Õ(d^{3.5}).

ABSTRACT

We present a fairly general framework for reducing $(\varepsilon, δ)$ differentially private (DP) statistical estimation to its non-private counterpart. As the main application of this framework, we give a polynomial time and $(\varepsilon,δ)$-DP algorithm for learning (unrestricted) Gaussian distributions in $\mathbb{R}^d$. The sample complexity of our approach for learning the Gaussian up to total variation distance $α$ is $\widetilde{O}(d^2/α^2 + d^2\sqrt{\ln(1/δ)}/α\varepsilon + d\ln(1/δ) / α\varepsilon)$ matching (up to logarithmic factors) the best known information-theoretic (non-efficient) sample complexity upper bound due to Aden-Ali, Ashtiani, and Kamath (ALT'21). In an independent work, Kamath, Mouzakis, Singhal, Steinke, and Ullman (arXiv:2111.04609) proved a similar result using a different approach and with $O(d^{5/2})$ sample complexity dependence on $d$. As another application of our framework, we provide the first polynomial time $(\varepsilon, δ)$-DP algorithm for robust learning of (unrestricted) Gaussians with sample complexity $\widetilde{O}(d^{3.5})$. In another independent work, Kothari, Manurangsi, and Velingker (arXiv:2112.03548) also provided a polynomial time $(\varepsilon, δ)$-DP algorithm for robust learning of Gaussians with sample complexity $\widetilde{O}(d^8)$.

Motivation & Objective

• To develop a general method for converting non-private statistical estimators into differentially private, polynomial-time algorithms.
• To address the open problem of efficient, private learning of unrestricted multivariate Gaussians under (ε,δ)-differential privacy.
• To provide the first polynomial-time, differentially private algorithm for robustly learning Gaussians when a fraction of samples are arbitrarily corrupted.
• To match or nearly match the best-known information-theoretic sample complexity bounds for private Gaussian learning, up to logarithmic factors.

Proposed method

• Introduce a general reduction technique that transforms any non-private statistical estimator into an (ε,δ)-differentially private algorithm with polynomial runtime.
• Apply the framework to private mean estimation using a private robust mean estimator (QUEScoreFilter) combined with noise injection via a masking mechanism.
• Use a private covariance estimation algorithm based on a private hypothesis selection method with a locally small cover for Gaussians.
• Combine private mean and covariance estimation to construct a complete private algorithm for learning multivariate Gaussians.
• Leverage concentration and privacy amplification techniques to ensure both privacy and utility guarantees under approximate differential privacy.
• Utilize a two-stage approach: first estimate the mean robustly under privacy constraints, then estimate the covariance using a private hypothesis selection mechanism with a carefully constructed cover.

Experimental results

Research questions

• RQ1 Can a general framework be designed to convert non-private statistical estimators into differentially private, polynomial-time algorithms?
• RQ2 Is it possible to achieve optimal sample complexity for private learning of unrestricted Gaussians in R^d using an efficient algorithm?
• RQ3 Can private robust learning of Gaussians be achieved efficiently under (ε,δ)-differential privacy when data is corrupted?
• RQ4 What is the tightest achievable sample complexity for private, efficient learning of multivariate Gaussians, and can it match non-private bounds up to logarithmic factors?

Key findings

• The proposed framework yields an (ε,δ)-DP algorithm for learning unrestricted Gaussians in R^d with sample complexity Õ(d²/α² + d²√(ln(1/δ))/(αε) + d ln(1/δ)/(αε)), matching the best-known non-private sample complexity up to logarithmic factors.
• The algorithm runs in polynomial time in d, 1/α, 1/ε, and log(1/δ), achieving efficient computation for high-dimensional Gaussians.
• The first polynomial-time, (ε,δ)-DP algorithm for robust learning of Gaussians is achieved with sample complexity Õ(d^{3.5}), significantly improving over prior work with Õ(d^8) dependence.
• The framework enables private robust mean estimation with error O(α√(log(1/α))) under (2ε, 4e^εδ)-DP, using a private robust mean estimator and noise injection.
• The robust covariance estimation component achieves utility guarantees under the same privacy budget, forming the backbone of the full private Gaussian learning pipeline.
• The sample complexity of the robust learning algorithm is dominated by the covariance estimation step, which is identified as a potential target for future optimization.