[Paper Review] A simple and practical algorithm for differentially private data release

Moritz Hardt, Katrina Ligett+1 more|arXiv (Cornell University)|Dec 21, 2010

Privacy-Preserving Technologies in DataComputer Science28 references303 citations

TL;DR

This paper introduces MWEM, a simple and practical differentially private algorithm that combines the Exponential Mechanism with Multiplicative Weights to release synthetic data with high accuracy. It achieves state-of-the-art theoretical guarantees and up to three orders of magnitude improvement in accuracy over prior methods on real-world datasets, while scaling to domains with thousands of attributes.

ABSTRACT

We present new theoretical results on differentially private data release useful with respect to any target class of counting queries, coupled with experimental results on a variety of real world data sets. Specifically, we study a simple combination of the multiplicative weights approach of [Hardt and Rothblum, 2010] with the exponential mechanism of [McSherry and Talwar, 2007]. The multiplicative weights framework allows us to maintain and improve a distribution approximating a given data set with respect to a set of counting queries. We use the exponential mechanism to select those queries most incorrectly tracked by the current distribution. Combing the two, we quickly approach a distribution that agrees with the data set on the given set of queries up to small error. The resulting algorithm and its analysis is simple, but nevertheless improves upon previous work in terms of both error and running time. We also empirically demonstrate the practicality of our approach on several data sets commonly used in the statistical community for contingency table release.

Motivation & Objective

• To develop a differentially private algorithm that balances utility and privacy with strong theoretical guarantees.
• To improve accuracy on real-world datasets compared to existing differentially private techniques, which often add excessive noise.
• To create a method that is simple to implement and use, requiring no deep mathematical understanding of query structures.
• To scale to high-dimensional datasets with up to 2^1000 domain size, overcoming limitations of prior methods.
• To enable consistent, downstream-compatible synthetic data generation that preserves statistical properties.

Proposed method

• The algorithm uses the Multiplicative Weights update rule to iteratively refine an approximating distribution over the data domain.
• It selects queries to measure using the Exponential Mechanism, prioritizing those most inconsistent with the current approximation.
• Each query is answered with differential privacy via the Laplace mechanism, ensuring ε-differential privacy.
• The method maintains a factored representation of the approximating distribution, enabling efficient computation on high-dimensional data.
• The algorithm dynamically partitions attributes into independent components when possible, reducing computational complexity.
• It integrates a scalable parallel implementation of Multiplicative Weights and supports datasets with up to 1000 attributes.

Experimental results

Research questions

• RQ1 Can a simple, general-purpose algorithm achieve better accuracy than existing differentially private methods on real-world datasets?
• RQ2 How does the integration of the Exponential Mechanism with Multiplicative Weights improve query accuracy and reduce noise?
• RQ3 To what extent can the algorithm scale to high-dimensional data with domains as large as 2^1000?
• RQ4 Does the factored representation of the approximating distribution significantly reduce running time without sacrificing accuracy?
• RQ5 Can the algorithm outperform specialized algorithms designed for specific query classes like range queries, contingency tables, and datacubes?

Key findings

• MWEM achieved up to three orders of magnitude improvement in accuracy over prior work on range queries for multiple real-world datasets under fixed privacy parameters.
• On contingency table release benchmarks, MWEM improved upon prior work across all tested statistical benchmarks.
• For datacube release, MWEM outperformed specialized algorithms designed for specific optimization criteria, demonstrating general-purpose superiority.
• The algorithm scaled to datasets with 2^1000 domain size, processing thousands of attributes, far exceeding the limits of prior methods.
• In experiments with the Adult dataset, the factored implementation ignored irrelevant attributes (added with p=0.1), maintaining low running time and maximum error.
• The total running time was dominated by query evaluation on the private dataset, while the factored MWEM component contributed negligibly to runtime.