[Paper Review] Graphical-model based estimation and inference for differential privacy

Ryan McKenna, Daniel Sheldon+1 more|arXiv (Cornell University)|Jan 26, 2019

Privacy-Preserving Technologies in Data35 citations

TL;DR

The paper introduces a graphical-model based approach to estimate data distributions from noisy marginal measurements under differential privacy, enabling scalable and accurate inference for private query workloads.

ABSTRACT

Many privacy mechanisms reveal high-level information about a data distribution through noisy measurements. It is common to use this information to estimate the answers to new queries. In this work, we provide an approach to solve this estimation problem efficiently using graphical models, which is particularly effective when the distribution is high-dimensional but the measurements are over low-dimensional marginals. We show that our approach is far more efficient than existing estimation techniques from the privacy literature and that it can improve the accuracy and scalability of many state-of-the-art mechanisms.

Motivation & Objective

• Motivate efficient estimation of high-dimensional data distributions from low-dimensional, privately noisy marginals.
• Develop a graphical-model framework that yields a maximum-entropy distribution matching observed marginals.
• Provide proximal estimation algorithms to recover marginals and graphical-model parameters without materializing the full contingency table.
• Enable accurate inference of new workload queries by efficient marginal inference on the learned graphical model.
• Demonstrate improvements when integrating the approach into state-of-the-art DP mechanisms.

Proposed method

• Represent the data distribution with a graphical model whose cliques correspond to measurement sets.
• Use a maximum-entropy principle to select a single distribution with the estimated marginals.
• Formulate estimation as convex optimization over the marginal polytope using marginals as sufficient statistics.
• Introduce two proximal algorithms (Algorithm 1 and Algorithm 2) that iteratively update clique marginals via a MARGINAL-ORACLE subroutine.
• Leverage belief propagation on junction trees as MARGINAL-ORACLE to perform tractable marginal inference.
• Apply the learned graphical model to answer workload queries without materializing the full contingency table.

Experimental results

Research questions

• RQ1 Can graphical models provide compact yet exact representations of high-dimensional data distributions under privacy noise?
• RQ2 How can one estimate marginals and recover a maximum-entropy distribution efficiently from privately noisy measurements?
• RQ3 Does integrating graphical-model based estimation improve accuracy and scalability of existing DP mechanisms for querying workloads?
• RQ4 What are practical algorithms with convergence guarantees to perform this estimation over marginal data?

Key findings

• Incorporating graphical-model estimation substantially improves accuracy for PrivBayes and DualQuery, with workload error reductions of 6x and 7x on Loans and Stroke datasets, and about 30% on Adult.
• DualQuery experiences improvements of 1.2x, 1.8x, 3.5x, and 4.4x across four datasets when using the graphical-model approach.
• MWEM and HDMM, which previously could not scale to the large workloads/datasets considered, become feasible with PGM-based estimation.
• The proposed proximal estimation algorithms come with convergence guarantees (O(1/√t) for general convex losses and O(1/t^2) for Lipschitz-gradient losses) and leverage a MARGINAL-ORACLE oracle for tractable marginal inference.
• The framework is compatible with existing private-query mechanisms and can be used as a building block to improve privacy-utility trade-offs without extra privacy loss.
• Experiments are conducted on Titanic, Adult, Loans, and Stroke datasets using a privacy budget of ε=1.0 (δ=0.001 for DualQuery).