[Paper Review] Mean Estimation with User-level Privacy under Data Heterogeneity

Rachel Cummings, Vitaly Feldman+2 more|arXiv (Cornell University)|Jul 28, 2023

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper proposes a differentially private mean estimation algorithm for heterogeneous user data, where users contribute varying numbers of samples from distinct distributions. It achieves asymptotically optimal error under user-level differential privacy by using a non-linear, truncation-based estimator that adapts to data heterogeneity and unknown variance, proving that privacy imposes no asymptotic cost in estimation error.

ABSTRACT

A key challenge in many modern data analysis tasks is that user data are heterogeneous. Different users may possess vastly different numbers of data points. More importantly, it cannot be assumed that all users sample from the same underlying distribution. This is true, for example in language data, where different speech styles result in data heterogeneity. In this work we propose a simple model of heterogeneous user data that allows user data to differ in both distribution and quantity of data, and provide a method for estimating the population-level mean while preserving user-level differential privacy. We demonstrate asymptotic optimality of our estimator and also prove general lower bounds on the error achievable in the setting we introduce.

Motivation & Objective

• Address the challenge of mean estimation in settings with user-level data heterogeneity, where users have varying sample sizes and distributions.
• Develop a differentially private algorithm that preserves user-level privacy while maintaining estimation accuracy despite distributional and sample-size heterogeneity.
• Demonstrate that the proposed estimator achieves asymptotically optimal error, with no additional cost from privacy constraints.
• Establish general lower bounds on estimation error in the heterogeneous, private setting to characterize fundamental limits.
• Provide theoretical guarantees for the estimator’s performance under mild assumptions on the underlying meta-distribution and data heterogeneity.

Proposed method

• Introduce a model where each user’s data is drawn from a Bernoulli distribution with mean $ p_i $, and $ p_i $ is itself a draw from an unknown meta-distribution $ \mathcal{D} $ on $[0,1]$.
• Propose a non-linear, differentially private estimator that adaptively truncates user contributions based on estimated variance and sample size.
• Use a two-stage estimation process: first estimate the global mean $ p $ and variance $ \sigma_p^2 $, then apply privacy-preserving truncation to individual user estimates.
• Apply privacy amplification via the advanced composition theorem and use concentration bounds for the meta-distribution and binomial sampling.
• Design truncation intervals $[\widehat{a}_i, \widehat{b}_i]$ that ensure privacy while maintaining accuracy, with widths bounded by a constant factor of the ideal interval.
• Leverage high-probability bounds on the meta-distribution and binomial sampling to ensure robustness and stability of the estimator.

Experimental results

Research questions

• RQ1 Can a differentially private mean estimator achieve asymptotically optimal error in the presence of user-level data heterogeneity?
• RQ2 What is the fundamental trade-off between privacy and estimation accuracy when user data distributions and sample sizes vary?
• RQ3 Does the price of user-level privacy vanish asymptotically in heterogeneous data settings, even when the global variance is unknown?
• RQ4 How can one design a private estimator that adapts to both the number of samples per user and the underlying distributional heterogeneity?
• RQ5 What are the tightest possible lower bounds on estimation error in this heterogeneous, private mean estimation setting?

Key findings

• The proposed differentially private estimator achieves variance $ \tilde{O}(\sigma_{\text{ideal}}^2) $, matching the non-private optimal variance up to logarithmic factors.
• The estimator is non-linear, demonstrating that linear estimators are insufficient for achieving optimal private mean estimation under data heterogeneity.
• Under mild assumptions, the asymptotic error of the private estimator matches the non-private lower bound, implying no asymptotic price for privacy.
• The width of the truncation intervals used in the estimator is bounded by a constant factor (6) of the ideal interval width, ensuring robustness.
• A novel lower bound technique is developed, showing that $ \sigma_{\text{ideal}}^2 $ is close to the best achievable variance under the given model and privacy constraints.
• The method remains effective even when the global variance $ \sigma_p^2 $ is unknown, with the estimator adapting through adaptive truncation and variance estimation.