[Paper Review] INSPECTRE: Privately Estimating the Unseen

Jayadev Acharya, Gautam Kamath+2 more|arXiv (Cornell University)|Feb 28, 2018

Privacy-Preserving Technologies in Data34 references5 citations

TL;DR

INSPECTRE proposes differentially private estimators for fundamental distribution properties—support size, support coverage, and entropy—by adapting state-of-the-art non-private methods with sensitivity analysis and Laplace noise. The key contribution is that the cost of privacy is nearly negligible, with only a small additive overhead proportional to $1/\alpha\varepsilon$, making private estimation feasible even in high-dimensional or sparse settings.

ABSTRACT

We develop differentially private methods for estimating various distributional properties. Given a sample from a discrete distribution $p$, some functional $f$, and accuracy and privacy parameters $α$ and $\varepsilon$, the goal is to estimate $f(p)$ up to accuracy $α$, while maintaining $\varepsilon$-differential privacy of the sample. We prove almost-tight bounds on the sample size required for this problem for several functionals of interest, including support size, support coverage, and entropy. We show that the cost of privacy is negligible in a variety of settings, both theoretically and experimentally. Our methods are based on a sensitivity analysis of several state-of-the-art methods for estimating these properties with sublinear sample complexities.

Motivation & Objective

• Address the challenge of estimating rare events in discrete distributions when data is scarce and contains sensitive information.
• Develop differentially private algorithms that maintain privacy while accurately estimating key distribution properties such as support size, support coverage, and entropy.
• Minimize the sample complexity required for private estimation, ensuring computational efficiency and practical applicability.
• Demonstrate that the cost of differential privacy is nearly negligible in realistic regimes where domain size $k$ is large and privacy budget $\varepsilon$ is moderate.

Proposed method

• Adapt non-private estimators from prior work (e.g., SGT for support coverage and others for entropy and support size) by analyzing their sensitivity to input changes.
• Apply Laplace mechanism to add noise proportional to the sensitivity of the estimator, ensuring $(\varepsilon, 0)$-differential privacy.
• Use a data-dependent parameter $r$ derived from the sample to control noise scale, based on theoretical insights from prior work.
• Implement privatized versions of existing sublinear-sample estimators, preserving their statistical efficiency while adding privacy guarantees.
• Calibrate noise using the sensitivity bound $2(1 + e^{r(t-1)})/\varepsilon$, which depends on the estimator’s structure and privacy parameter $\varepsilon$.
• Validate performance through extensive experiments on synthetic data, US Census data, and Shakespeare’s Hamlet, comparing with non-private baselines.

Experimental results

Research questions

• RQ1 What is the minimal sample complexity required to privately estimate distribution properties like support size, support coverage, and entropy?
• RQ2 How does the cost of differential privacy scale with respect to accuracy $\alpha$, privacy budget $\varepsilon$, and domain size $k$?
• RQ3 Can private estimation of complex distribution properties be achieved with only a small additive overhead compared to non-private methods?
• RQ4 In what regimes does the privacy cost become negligible, and when does it significantly degrade performance?
• RQ5 How do the privatized estimators perform on real-world datasets with high sparsity or oversampling, such as census data or literary texts?

Key findings

• The sample complexity for private estimation of entropy incurs an additive cost of $O(1/\alpha\varepsilon)$, which is nearly optimal and independent of the domain size $k$.
• For large $k$, the privacy overhead is $o(1)$ in the regime where $k \gg 1/\alpha\varepsilon$, meaning privacy adds negligible cost to the inherent sample complexity.
• Experimental results on synthetic data show that the RMSE of the private estimator is nearly indistinguishable from the non-private SGT estimator for $\varepsilon = 1, 2, 10$.
• On real-world datasets like the 2000 US Census and Shakespeare’s Hamlet, the private estimator achieves performance nearly identical to the non-private baseline, even at $\varepsilon = 0.5$.
• For small support sizes (e.g., $k \approx 100$), the cost of privacy becomes significant due to the proximity of neighboring datasets with very different support sizes, creating a fundamental trade-off.
• Theoretical lower bounds confirm that the upper bounds on sample complexity are almost tight, validating the efficiency of the proposed approach.