[Paper Review] Private Convex Optimization via Exponential Mechanism

Sivakanth Gopi, Yin Tat Lee+1 more|arXiv (Cornell University)|Mar 1, 2022

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper introduces a regularized exponential mechanism for private convex optimization that achieves optimal excess empirical and population risk under $(\varepsilon,\delta)$-differential privacy. By adding an $\ell_2^2$ regularizer and sampling from a strongly log-concave distribution, the method enables efficient optimization with $\widetilde{O}(n\min(d,n))$ function queries, matching a nearly tight lower bound.

ABSTRACT

In this paper, we study private optimization problems for non-smooth convex functions $F(x)=\mathbb{E}_i f_i(x)$ on $\mathbb{R}^d$. We show that modifying the exponential mechanism by adding an $\ell_2^2$ regularizer to $F(x)$ and sampling from $π(x)\propto \exp(-k(F(x)+μ\|x\|_2^2/2))$ recovers both the known optimal empirical risk and population loss under $(ε,δ)$-DP. Furthermore, we show how to implement this mechanism using $\widetilde{O}(n \min(d, n))$ queries to $f_i(x)$ for the DP-SCO where $n$ is the number of samples/users and $d$ is the ambient dimension. We also give a (nearly) matching lower bound $\widetildeΩ(n \min(d, n))$ on the number of evaluation queries. Our results utilize the following tools that are of independent interest: (1) We prove Gaussian Differential Privacy (GDP) of the exponential mechanism if the loss function is strongly convex and the perturbation is Lipschitz. Our privacy bound is \emph{optimal} as it includes the privacy of Gaussian mechanism as a special case and is proved using the isoperimetric inequality for strongly log-concave measures. (2) We show how to sample from $\exp(-F(x)-μ\|x\|^2_2/2)$ for $G$-Lipschitz $F$ with $η$ error in total variation (TV) distance using $\widetilde{O}((G^2/μ) \log^2(d/η))$ unbiased queries to $F(x)$. This is the first sampler whose query complexity has \emph{polylogarithmic dependence} on both dimension $d$ and accuracy $η$.

Motivation & Objective

• To develop a differentially private optimization framework for non-smooth convex functions $F(x) = \mathbb{E}_i f_i(x)$ with optimal utility under $(\varepsilon,\delta)$-differential privacy.
• To close the gap between the query complexity of exponential mechanism and noisy gradient descent in private convex optimization by introducing a regularized variant.
• To establish tight upper and lower bounds on the number of function queries required for private optimization, achieving nearly optimal efficiency.
• To develop a new sampling algorithm with polylogarithmic dependence on dimension $d$ and accuracy $\eta$, enabling efficient implementation of the regularized mechanism.

Proposed method

• The method modifies the exponential mechanism by adding an $\ell_2^2$ regularizer, sampling from $\pi(x) \propto \exp(-k(F(x) + \mu\|x\|_2^2/2))$ to ensure strong convexity and improved privacy guarantees.
• It proves Gaussian Differential Privacy (GDP) for the regularized mechanism under strong convexity and Lipschitz perturbations, using isoperimetric inequalities for strongly log-concave measures.
• It introduces a novel sampling algorithm that achieves $\eta$-accuracy in total variation distance using $\widetilde{O}((G^2/\mu)\log^2(d/\eta))$ unbiased queries to $F(x)$, with polylogarithmic dependence on $d$ and $\eta$.
• The framework achieves optimal excess empirical risk $O(GDd/(n\varepsilon))$ under $(\varepsilon,0)$-DP and $O(GD\sqrt{d\log(1/\delta)}/(n\varepsilon))$ under $(\varepsilon,\delta)$-DP, matching known lower bounds.
• It establishes a nearly matching lower bound of $\widetilde{\Omega}(n\min(d,n))$ on query complexity, showing the method is nearly optimal in terms of function evaluations.

Experimental results

Research questions

• RQ1 Can the exponential mechanism be regularized to achieve optimal privacy-utility tradeoffs in non-smooth private convex optimization?
• RQ2 What is the optimal query complexity for sampling from a regularized exponential mechanism under differential privacy?
• RQ3 Can a sampling algorithm be designed with polylogarithmic query complexity in dimension $d$ and accuracy $\eta$ for $G$-Lipschitz convex functions?
• RQ4 Does the regularized exponential mechanism achieve both optimal empirical and population risk under $(\varepsilon,\delta)$-DP?
• RQ5 What is the information-theoretic lower bound on the number of function queries required for private convex optimization?

Key findings

• The regularized exponential mechanism achieves optimal excess empirical risk $O(GDd/(n\varepsilon))$ under $(\varepsilon,0)$-DP, matching the best-known upper bound.
• Under $(\varepsilon,\delta)$-DP, the method achieves excess population risk $O(GD\sqrt{d\log(1/\delta)}/(n\varepsilon))$, which is optimal and matches the performance of noisy gradient descent.
• The proposed sampling algorithm achieves $\eta$-accuracy in total variation distance using $\widetilde{O}((G^2/\mu)\log^2(d/\eta))$ queries, with polylogarithmic dependence on $d$ and $\eta$, a significant improvement over prior methods.
• The method's query complexity is $\widetilde{O}(n\min(d,n))$, which matches a nearly tight lower bound of $\widetilde{\Omega}(n\min(d,n))$ on the number of function evaluations.
• A nearly matching lower bound of $\widetilde{\Omega}(n\min(d,n))$ is established, showing the method's efficiency is nearly optimal in terms of query complexity.