[Paper Review] The Geometry of Differential Privacy: the Sparse and Approximate Cases

Aleksandar Nikolov, Kunal Talwar+1 more|arXiv (Cornell University)|Dec 3, 2012

Privacy-Preserving Technologies in DataComputer Science68 references98 citations

TL;DR

This paper presents an $O(\log^2 d)$-approximation mechanism for $(\varepsilon,\delta)$-differentially private linear query release using correlated Gaussian noise, achieving near-optimal accuracy. It further introduces a polylogarithmic approximation for sparse databases where $d > n$, combining Gaussian noise with $\ell_1$-regularized regression, improving error bounds for counting queries to $\tilde{O}(\sqrt{n})$ per query.

ABSTRACT

In this work, we study trade-offs between accuracy and privacy in the context of linear queries over histograms. This is a rich class of queries that includes contingency tables and range queries, and has been a focus of a long line of work. For a set of $d$ linear queries over a database $x \in \R^N$, we seek to find the differentially private mechanism that has the minimum mean squared error. For pure differential privacy, an $O(\log^2 d)$ approximation to the optimal mechanism is known. Our first contribution is to give an $O(\log^2 d)$ approximation guarantee for the case of $(\eps,δ)$-differential privacy. Our mechanism is simple, efficient and adds correlated Gaussian noise to the answers. We prove its approximation guarantee relative to the hereditary discrepancy lower bound of Muthukrishnan and Nikolov, using tools from convex geometry. We next consider this question in the case when the number of queries exceeds the number of individuals in the database, i.e. when $d > n riangleq \|x\|_1$. It is known that better mechanisms exist in this setting. Our second main contribution is to give an $(\eps,δ)$-differentially private mechanism which is optimal up to a $\polylog(d,N)$ factor for any given query set $A$ and any given upper bound $n$ on $\|x\|_1$. This approximation is achieved by coupling the Gaussian noise addition approach with a linear regression step. We give an analogous result for the $\eps$-differential privacy setting. We also improve on the mean squared error upper bound for answering counting queries on a database of size $n$ by Blum, Ligett, and Roth, and match the lower bound implied by the work of Dinur and Nissim up to logarithmic factors. The connection between hereditary discrepancy and the privacy mechanism enables us to derive the first polylogarithmic approximation to the hereditary discrepancy of a matrix $A$.

Motivation & Objective

• To close the gap in approximation guarantees for $(\varepsilon,\delta)$-differentially private mechanisms in the context of linear queries over histograms.
• To address the challenge of query accuracy when the number of queries $d$ exceeds the number of individuals $n$ (sparse regime), where standard lower bounds no longer apply.
• To improve error bounds for counting queries under differential privacy, particularly in the sparse regime where $d > n$.
• To establish a connection between hereditary discrepancy and differentially private mechanisms, enabling polylogarithmic approximation of discrepancy.
• To design efficient, simple mechanisms that achieve near-optimal error under both pure and approximate differential privacy.

Proposed method

• Uses correlated Gaussian noise addition to construct an $(\varepsilon,\delta)$-differentially private mechanism with $O(\log^2 d)$ approximation to the optimal mechanism.
• Applies tools from convex geometry to prove the approximation guarantee relative to the hereditary discrepancy lower bound from [MN12].
• Combines Gaussian noise addition with $\ell_1$-regularized regression over the $\ell_1$-ball to achieve error within $\operatorname{polylog}(d,N)$ of the optimal in the sparse regime.
• Employs a constructive approach to SDP feasibility via sampling and truncation, ensuring empirical estimates satisfy constraints up to a constant factor.
• Leverages the fact that differential privacy implies bounded variance in inverse query responses, enabling lower bounds on the covariance matrix of the mechanism’s output.
• Uses iterative refinement to construct a feasible solution to a semidefinite program (SDP) by combining positive semidefinite matrices derived from query subproblems.

Experimental results

Research questions

• RQ1 Can we achieve a polylogarithmic approximation ratio for $(\varepsilon,\delta)$-differentially private mechanisms in the general case of linear queries?
• RQ2 What is the optimal trade-off between accuracy and privacy when $d > n$, and can we design mechanisms that outperform existing bounds in this sparse regime?
• RQ3 Can the hereditary discrepancy lower bound be used to derive approximation guarantees for differentially private mechanisms?
• RQ4 How can we improve the error bound for counting queries under differential privacy beyond the $\tilde{O}(n^{2/3})$ bound from prior work?
• RQ5 Is it possible to construct a differentially private mechanism with $\ell_2^2$ error within $\operatorname{polylog}(d,N)$ of the optimal in the sparse setting?

Key findings

• The proposed $(\varepsilon,\delta)$-differentially private mechanism achieves an $O(\log^2 d)$ approximation to the optimal mechanism, matching the best-known bound for pure differential privacy.
• For the sparse regime where $d > n$, the mechanism achieves mean squared error within $\operatorname{polylog}(d,N)$ of the optimal, significantly improving over prior bounds.
• For arbitrary counting queries, the mechanism achieves expected error $\tilde{O}(\sqrt{n})$ per query, improving on the $\tilde{O}(n^{2/3})$ bound from [BLR08] and matching the lower bound from [DN03] up to logarithmic factors.
• The paper establishes the first polylogarithmic approximation to the hereditary discrepancy of a matrix $A$, via the connection to differentially private mechanisms.
• The mechanism is simple and efficient, relying on correlated Gaussian noise and $\ell_1$-regularized regression, with a constructive SDP-based analysis.
• The analysis shows that differential privacy implies a lower bound on the variance of inverse query responses, enabling the construction of feasible solutions to semidefinite programs.