[Paper Review] The Cost of Privacy: Optimal Rates of Convergence for Parameter Estimation with Differential Privacy

Tommaso Cai, Yichen Wang+1 more|arXiv (Cornell University)|Feb 12, 2019

Privacy-Preserving Technologies in Data43 references48 citations

TL;DR

The paper establishes minimax lower bounds and matches them with differential privacy algorithms for mean estimation and linear regression, characterizing the optimal trade-off between privacy (epsilon, delta) and statistical accuracy in both low- and high-dimensional settings.

ABSTRACT

Privacy-preserving data analysis is a rising challenge in contemporary statistics, as the privacy guarantees of statistical methods are often achieved at the expense of accuracy. In this paper, we investigate the tradeoff between statistical accuracy and privacy in mean estimation and linear regression, under both the classical low-dimensional and modern high-dimensional settings. A primary focus is to establish minimax optimality for statistical estimation with the $(\varepsilon,δ)$-differential privacy constraint. To this end, we find that classical lower bound arguments fail to yield sharp results, and new technical tools are called for. By refining the "tracing adversary" technique for lower bounds in the theoretical computer science literature, we formulate a general lower bound argument for minimax risks with differential privacy constraints, and apply this argument to high-dimensional mean estimation and linear regression problems. We also design computationally efficient algorithms that attain the minimax lower bounds up to a logarithmic factor. In particular, for the high-dimensional linear regression, a novel private iterative hard thresholding pursuit algorithm is proposed, based on a privately truncated version of stochastic gradient descent. The numerical performance of these algorithms is demonstrated by simulation studies and applications to real data containing sensitive information, for which privacy-preserving statistical methods are necessary.

Motivation & Objective

• Motivate and formalize the cost of privacy under (epsilon, delta)-differential privacy in fundamental estimation problems.
• Derive minimax lower bounds for mean estimation and linear regression under privacy constraints.
• Design differentially private algorithms that achieve these lower bounds up to logarithmic factors.
• Provide theoretical and empirical validation of privacy-accuracy tradeoffs in both low- and high-dimensional settings.

Proposed method

• Define the cost of privacy via minimax risk under (epsilon, delta)-DP for mean estimation and linear regression.
• Refine tracing adversary techniques to obtain sharp DP lower bounds for low- and high-dimensional problems.
• Construct DP algorithms (Gaussian/noise-perturbed mechanisms, private iterative methods) that achieve the lower bounds up to log factors.
• Introduce a private sparse mean estimation approach using a peeling mechanism to select coordinates privately.
• Analyze convergence rates and derive bounds like tilde{O}((d^2 log(1/delta))/ (n^2 epsilon^2)) in appropriate regimes.

Experimental results

Research questions

• RQ1 What is the minimax risk for estimating mean vectors and regression coefficients under (epsilon, delta)-DP in low- and high-dimensional settings?
• RQ2 Can differential privacy algorithms attain the corresponding minimax lower bounds, establishing optimal rates of convergence?
• RQ3 How does sparsity affect the cost of privacy in high-dimensional mean estimation and regression?
• RQ4 What are practical private algorithms that achieve these optimal rates and how do they perform empirically?
• RQ5 How does the privacy cost compare to classical statistical risk across regimes of n, d, and s*?

Key findings

• Lower bounds show the cost of privacy dominates statistical risk when (d log(1/delta))/ (n epsilon^2) is large, with bounds like Omega(d/n + d^2 log(1/delta)/ (n^2 epsilon^2)) for mean estimation.
• New high-dimensional lower bounds for mean estimation and linear regression under DP, with rates involving (s log d)^2 / (n^2 epsilon^2).
• Propose DP algorithms achieving the lower bounds up to logarithmic factors, including a noisy gradient descent for regression with tilde{O}(d^2 log(1/delta)/(n^2 epsilon^2)) convergence, and a private peeling-based sparse mean estimator achieving the sparse lower bound.
• For high-dimensional sparse estimation, the DP rates depend on (s log d)^2 and are nearly optimal up to log factors, signaling feasibility of DP in high dimensions.
• Provide numerical simulations and real-data applications illustrating the privacy-accuracy tradeoffs.