[Paper Review] Lossless Compression of Efficient Private Local Randomizers

Vitaly Feldman, Kunal Talwar|arXiv (Cornell University)|Feb 24, 2021

Privacy-Preserving Technologies in Data45 references4 citations

TL;DR

This paper presents a general method to losslessly compress locally differentially private (LDP) randomizers using pseudo-random generators under standard cryptographic assumptions, enabling communication costs reduced to the size of a PRG seed while preserving privacy and utility. The approach achieves up to 5× lower error than prior compressed LDP algorithms in high-privacy regimes (ε > 1).

ABSTRACT

Locally Differentially Private (LDP) Reports are commonly used for collection of statistics and machine learning in the federated setting. In many cases the best known LDP algorithms require sending prohibitively large messages from the client device to the server (such as when constructing histograms over large domain or learning a high-dimensional model). This has led to significant efforts on reducing the communication cost of LDP algorithms. At the same time LDP reports are known to have relatively little information about the user's data due to randomization. Several schemes are known that exploit this fact to design low-communication versions of LDP algorithm but all of them do so at the expense of a significant loss in utility. Here we demonstrate a general approach that, under standard cryptographic assumptions, compresses every efficient LDP algorithm with negligible loss in privacy and utility guarantees. The practical implication of our result is that in typical applications the message can be compressed to the size of the server's pseudo-random generator seed. More generally, we relate the properties of an LDP randomizer to the power of a pseudo-random generator that suffices for compressing the LDP randomizer. From this general approach we derive low-communication algorithms for the problems of frequency estimation and high-dimensional mean estimation. Our algorithms are simpler and more accurate than existing low-communication LDP algorithms for these well-studied problems.

Motivation & Objective

• Address the high communication cost of efficient LDP algorithms, especially for large-domain or high-dimensional data.
• Overcome the limitations of prior compression techniques that degrade utility or increase ε, particularly in the ε > 1 regime.
• Develop a general, lossless compression framework for any efficient LDP randomizer that maintains both privacy and utility guarantees.
• Enable practical deployment of high-utility LDP mechanisms in federated settings with constrained client communication.
• Demonstrate the method’s effectiveness on frequency estimation and high-dimensional mean estimation, outperforming existing low-communication approaches.

Proposed method

• Use a data-independent reference distribution ρ that is ε-close to the output distribution of any input under the LDP randomizer R.
• Replace true random bits in R with pseudo-random bits generated from a seed using a strong pseudo-random generator (PRG).
• Perform rejection sampling relative to pseudo-random samples from ρ to emulate R(x) while preserving privacy.
• Ensure privacy preservation by requiring the PRG to fool a computationally bounded test that compares the density ratio of R(x) to ρ.
• Leverage standard cryptographic assumptions (e.g., existence of exponentially strong PRGs) to ensure computational indistinguishability from true randomness.
• Apply the method to specific LDP mechanisms like PrivUnit and PrivHS, optimizing parameters to minimize variance proxies for improved utility.

Experimental results

Research questions

• RQ1 Can we compress the output of any efficient LDP randomizer without losing privacy or utility?
• RQ2 What cryptographic assumptions are sufficient to ensure that pseudo-random compression preserves ε-differential privacy?
• RQ3 How does the compression technique perform in practice on well-known LDP problems like frequency estimation and high-dimensional mean estimation?
• RQ4 Can the compression method achieve better utility than existing low-communication LDP algorithms, especially in the ε > 1 regime?
• RQ5 What is the impact of splitting the privacy budget across multiple runs on the error and computational cost of the compressed mechanism?

Key findings

• The proposed compression method preserves both privacy and utility with only a negligible loss, under standard cryptographic assumptions.
• For d=2000 and ε=8, the proxy variance is accurate to within 3.5%, validating the use of norm proxies as accurate surrogates for true error.
• PrivUnitOptimized achieves up to 5× lower expected squared error than SQKR and 2.5× lower than standard PrivUnit, significantly outperforming prior compressed LDP algorithms.
• Splitting ε=8 into two ε=4 runs increases expected squared error by approximately 2×, indicating a trade-off between computation and accuracy.
• The method is general and can be applied to any future efficient LDP randomizer without re-derivation, enabling future gains in utility and communication efficiency.
• The compression reduces message size to the length of the PRG seed, enabling practical deployment in bandwidth-constrained federated learning and data collection systems.