[Paper Review] Breaking the Communication-Privacy-Accuracy Trilemma

Weining Chen, Peter Kairouz+1 more|arXiv (Cornell University)|Jul 22, 2020

Privacy-Preserving Technologies in DataComputer Science53 references32 citations

TL;DR

The paper introduces new encoding and decoding schemes that jointly optimize local differential privacy and communication constraints to achieve near-optimal accuracy for mean and frequency estimation, breaking the trade-off among privacy, communication, and accuracy under ε-LDP and b-bit limits.

ABSTRACT

Two major challenges in distributed learning and estimation are 1) preserving the privacy of the local samples; and 2) communicating them efficiently to a central server, while achieving high accuracy for the end-to-end task. While there has been significant interest in addressing each of these challenges separately in the recent literature, treatments that simultaneously address both challenges are still largely missing. In this paper, we develop novel encoding and decoding mechanisms that simultaneously achieve optimal privacy and communication efficiency in various canonical settings. In particular, we consider the problems of mean estimation and frequency estimation under $\varepsilon$-local differential privacy and $b$-bit communication constraints. For mean estimation, we propose a scheme based on Kashin's representation and random sampling, with order-optimal estimation error under both constraints. For frequency estimation, we present a mechanism that leverages the recursive structure of Walsh-Hadamard matrices and achieves order-optimal estimation error for all privacy levels and communication budgets. As a by-product, we also construct a distribution estimation mechanism that is rate-optimal for all privacy regimes and communication constraints, extending recent work that is limited to $b=1$ and $\varepsilon=O(1)$. Our results demonstrate that intelligent encoding under joint privacy and communication constraints can yield a performance that matches the optimal accuracy achievable under either constraint alone.

Motivation & Objective

• Motivate and formalize the joint privacy-communication-accuracy trade-off in distributed learning and estimation.
• Provide schemes that attain optimal or near-optimal estimation error under ε-LDP and b-bit communication constraints across canonical tasks.
• Characterize when one constraint dominates and how the less stringent constraint can be satisfied for free.
• Demonstrate universal schemes applicable to both mean and distribution/frequency estimation with shared randomness considerations.

Proposed method

• Develop a public-coin scheme for mean estimation based on Kashin’s representation and random sampling (SQKR) to achieve order-optimal ℓ2 error under ε-LDP and b-bit constraints.
• Pre-process data with Kashin’s representation to distribute information evenly across coefficients, enabling robust quantization and privatization.
• Quantize, subsample, and privatize the data using a 2^k-RR mechanism to send k-bit reports, then reconstruct unbiased estimators at the server.
• For statistical mean estimation, provide a variant that avoids shared randomness by deterministically partitioning coordinates (still achieving optimal error under ε-LDP and b-bit constraints).
• Introduce a Recursive Hadamard Response (RHR) scheme for frequency estimation that exploits the recursive Hadamard structure to achieve order-optimal errors for all privacy and communication budgets.
• Demonstrate that the dominant constraint determines the error, and the other constraint can be satisfied for free, yielding practical schemes with O(n+d log d) decoding complexity in the frequency-estimation setting.

Experimental results

Research questions

• RQ1 What are the fundamental estimation-error limits under joint ε-LDP and b-bit communication constraints for canonical tasks (mean, frequency, and distribution estimation)?
• RQ2 Can encoding schemes that leverage Kashin’s representation or recursive Hadamard structures achieve order-optimal performance across all privacy and communication budgets?
• RQ3 To what extent can the less stringent constraint be satisfied for free when the other constraint dominates the error?
• RQ4 How do shared randomness requirements affect the practicality and optimality of proposed schemes in statistical versus distributional settings?

Key findings

• For mean estimation, r_ME(ℓ2, ε, b) = Θ(d/n · min(ε^2, ε, b)); the SQKR scheme achieves this order and is information-theoretically optimal under certain regimes.
• Kashin’s representation spreads information evenly across coefficients, enabling unbiased private estimates with low-variance reconstruction and improved ℓ2 error under joint constraints.
• For frequency estimation, r_FE(ℓ2) = Θ(d/(n min{e^ε, (e^ε−1)^2, 2^b, d})) and r_FE(ℓ1) = Θ(d/(√(n min{e^ε, (e^ε−1)^2, 2^b, d}))); RHR achieves order-optimality across ε and b with efficient decoding (O(n + d log d)).
• The Recursive Hadamard Response (RHR) extends to distribution estimation without shared randomness, achieving order-optimal ℓ1 and ℓ2 errors for all privacy regimes and budgets.
• The results show that the fundamental trade-off is governed by the stricter constraint, and the less stringent one can be satisfied for free, explaining why small- or single-bit budgets can suffice in high-privacy regimes.
• The schemes achieve matching information-theoretic lower bounds in the studied settings, validating their optimality.