[Paper Review] Near Instance-Optimality in Differential Privacy

Hilal Asi, John C. Duchi|arXiv (Cornell University)|May 16, 2020

Privacy-Preserving Technologies in Data36 references4 citations

TL;DR

This paper introduces instance-optimality in differential privacy by defining local minimax risk and unbiased mechanism bounds, showing that the local modulus of continuity of the estimand fully determines optimal error. It proposes inverse sensitivity mechanisms that adapt to data instance and uniformly outperform smooth sensitivity and Laplace mechanisms for continuous functions and robust estimands like median and regression.

ABSTRACT

We develop two notions of instance optimality in differential privacy, inspired by classical statistical theory: one by defining a local minimax risk and the other by considering unbiased mechanisms and analogizing the Cramer-Rao bound, and we show that the local modulus of continuity of the estimand of interest completely determines these quantities. We also develop a complementary collection mechanisms, which we term the inverse sensitivity mechanisms, which are instance optimal (or nearly instance optimal) for a large class of estimands. Moreover, these mechanisms uniformly outperform the smooth sensitivity framework on each instance for several function classes of interest, including real-valued continuous functions. We carefully present two instantiations of the mechanisms for median and robust regression estimation with corresponding experiments.

Motivation & Objective

• To develop instance-specific notions of optimality in differential privacy that reflect data-dependent difficulty rather than worst-case bounds.
• To establish lower bounds for private mechanisms using local minimax risk and unbiased estimation, analogous to classical Cramér-Rao bounds.
• To design a new class of mechanisms—inverse sensitivity mechanisms—that adapt to the data instance and achieve near-instance optimality.
• To demonstrate that these mechanisms uniformly outperform existing frameworks like smooth sensitivity and Laplace mechanisms across function classes.
• To extend the framework to continuous estimands such as median and robust regression with theoretical and empirical validation.

Proposed method

• Defines instance-specific lower bounds using a private local minimax framework, where the local modulus of continuity of the function at a dataset determines the error floor.
• Introduces the inverse sensitivity mechanism as a private exponential mechanism using the inverse of local sensitivity: $\mathbb{P}(M(\mathbf{x})=t) \propto \exp(-\mathsf{len}_f(\mathbf{x};t)\varepsilon/2)$.
• For continuous estimands, constructs a continuous analog of inverse sensitivity by defining $\mathsf{len}_f(\mathbf{x};t)$ as the minimum Hamming distance to a dataset where $f(\mathbf{x}')=t$.
• Uses a change-of-variables and rotational symmetry to sample from $\pi(t) \propto \exp(-\|A t\|)$, enabling efficient sampling in high dimensions.
• Applies Taylor expansion and error bounds to compare the target density $\pi$ with a quadratic approximation $q$, showing their ratio is bounded away from zero.
• Empirically validates the mechanism on median and robust regression, showing improved accuracy over Laplace and smooth sensitivity mechanisms.

Experimental results

Research questions

• RQ1 Can instance-specific lower bounds in differential privacy be defined such that they reflect the true difficulty of private estimation at a given dataset?
• RQ2 To what extent can private mechanisms adapt to the local structure of a dataset, rather than relying on worst-case global sensitivity?
• RQ3 Is there a mechanism class that achieves near-instance optimality across diverse function classes, including continuous and robust estimands?
• RQ4 How does the inverse sensitivity mechanism compare to smooth sensitivity and Laplace mechanisms in terms of error and adaptivity?
• RQ5 Can the theoretical optimality of inverse sensitivity mechanisms be extended to continuous output spaces with provable performance guarantees?

Key findings

• The local modulus of continuity of the estimand completely determines the instance-specific minimax risk and unbiased estimation lower bounds.
• The inverse sensitivity mechanism is more likely to return the true value $f(\mathbf{x})$ than any other $\varepsilon$-differentially private mechanism at the same dataset.
• For $\mathbb{R}$-valued continuous functions, the inverse sensitivity mechanism uniformly outperforms both Laplace and smooth sensitivity mechanisms on every instance.
• In median estimation, the inverse sensitivity mechanism achieves significantly lower error than Laplace and smooth sensitivity, especially in sparse or skewed data.
• For robust regression, the mechanism maintains high accuracy even under contamination, outperforming baseline methods due to data-adaptive noise scaling.
• Theoretical analysis confirms that the ratio of the target density $\pi$ to its quadratic approximation $q$ is bounded away from zero, ensuring near-optimality.