[Paper Review] Exact Inference with Approximate Computation for Differentially Private Data via Perturbations

Ruobin Gong|arXiv (Cornell University)|Sep 26, 2019

Statistical Methods and Bayesian Inference4 citations

TL;DR

This paper proposes a modular framework to enable exact Bayesian and likelihood inference from differentially private data using approximate computation methods—approximate Bayesian computation (ABC) and Monte Carlo EM—by aligning their tuning parameters with the underlying privacy perturbation mechanism. The key contribution is that, up to Monte Carlo error, the inference is exact with respect to the joint model of the analyst’s statistical model and the curator’s privacy mechanism.

ABSTRACT

This paper discusses how two classes of approximate computation algorithms can be adapted, in a modular fashion, to achieve exact statistical inference from differentially private data products. Considered are approximate Bayesian computation for Bayesian inference, and Monte Carlo Expectation-Maximization for likelihood inference. Up to Monte Carlo error, inference from these algorithms is exact with respect to the joint specification of both the analyst's original data model, and the curator's differential privacy mechanism. Highlighted is a duality between approximate computation on exact data, and exact computation on approximate data, which can be leveraged by a well-designed computational procedure for statistical inference.

Motivation & Objective

• To enable exact statistical inference from differentially private data products without requiring analytical re-derivation of the joint posterior or likelihood.
• To address the challenge of conducting valid inference when data have been perturbed for differential privacy, especially when the privacy mechanism is complex or unknown.
• To establish a duality between approximate computation on exact data and exact computation on approximate data, leveraging transparency of the privacy mechanism.
• To provide a computationally feasible and theoretically sound approach that maintains statistical validity under differential privacy.
• To ensure that inference remains valid even when the privacy mechanism involves clamping or post-processing, provided the mechanism remains transparent.

Proposed method

• Adapts approximate Bayesian computation (ABC) by using a kernel and bandwidth that exactly match the privacy perturbation mechanism used in data publication.
• Employs a rejection sampling algorithm in ABC where synthetic data are accepted based on proximity to the privatized data, with the kernel tied to the privacy mechanism’s noise distribution.
• Applies importance sampling within a Monte Carlo EM (MCEM) framework to estimate maximum likelihood parameters from privatized data, using the privacy mechanism’s density as a proposal.
• Ensures that the posterior or likelihood approximation is exact with respect to the joint model of the original data model and the differential privacy mechanism.
• Relies on the transparency of the privacy mechanism’s conditional density $\eta_{\text{dp}}(\boldsymbol{s}_{\text{dp}} \mid \cdot)$, which must be analytically tractable for the method to work.
• Designs the algorithms so that the tuning parameters (e.g., kernel bandwidth, importance weights) are explicitly aligned with the privacy mechanism, enabling exact inference despite data perturbation.

Experimental results

Research questions

• RQ1 Can approximate computation methods like ABC and MCEM be adapted to produce exact statistical inference from differentially private data?
• RQ2 How can the privacy mechanism’s perturbation structure be leveraged to ensure that inference remains exact with respect to the joint model of the data model and privacy mechanism?
• RQ3 What conditions on the privacy mechanism (e.g., transparency, form of noise) are necessary for exact inference using these approximate computation methods?
• RQ4 How does the duality between approximate computation on exact data and exact computation on approximate data inform the design of privacy-aware inference procedures?
• RQ5 What are the computational and statistical limitations of these methods when the privacy mechanism involves clamping or post-processing?

Key findings

• The proposed ABC algorithm produces independent and identically distributed samples from the true posterior distribution when the kernel and bandwidth are matched to the privacy perturbation mechanism.
• The MCEM-based approach using importance sampling yields exact likelihood inference under the joint model, provided the privacy mechanism’s density is transparent and analytically available.
• The method achieves exact inference up to Monte Carlo error, meaning the resulting inference is statistically valid with respect to both the original data model and the curator’s privacy mechanism.
• The approach is modular and does not require re-deriving the joint model analytically, enabling reuse of existing ABC and MCEM implementations for differentially private data.
• The performance of ABC is sensitive to prior-data conflict; low acceptance rates occur when the prior is diffuse or in conflict with the observed data, requiring more sampling effort in high-posterior-density regions.
• The feasibility of the method depends critically on the transparency of the privacy mechanism; clamping or post-processing that obscures the mechanism’s density can break the method’s validity.