[Paper Review] Empirical Risk Minimization in the Non-interactive Local Model of Differential Privacy

Di Wang, Marco Gaboardi+2 more|arXiv (Cornell University)|Nov 11, 2020

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper proposes efficient non-interactive locally differentially private algorithms for Empirical Risk Minimization (ERM) in high-dimensional settings. By leveraging Bernstein polynomial approximation for smooth loss functions and inner product polynomial approximation for generalized linear models, the authors achieve sample complexity that is polynomial in the error parameter α and linear in dimension p, overcoming the prior exponential dependence on p.

ABSTRACT

In this paper, we study the Empirical Risk Minimization (ERM) problem in the non-interactive Local Differential Privacy (LDP) model. Previous research on this problem \citep{smith2017interaction} indicates that the sample complexity, to achieve error $α$, needs to be exponentially depending on the dimensionality $p$ for general loss functions. In this paper, we make two attempts to resolve this issue by investigating conditions on the loss functions that allow us to remove such a limit. In our first attempt, we show that if the loss function is $(\infty, T)$-smooth, by using the Bernstein polynomial approximation we can avoid the exponential dependency in the term of $α$. We then propose player-efficient algorithms with $1$-bit communication complexity and $O(1)$ computation cost for each player. The error bound of these algorithms is asymptotically the same as the original one. With some additional assumptions, we also give an algorithm which is more efficient for the server. In our second attempt, we show that for any $1$-Lipschitz generalized linear convex loss function, there is an $(ε, δ)$-LDP algorithm whose sample complexity for achieving error $α$ is only linear in the dimensionality $p$. Our results use a polynomial of inner product approximation technique. Finally, motivated by the idea of using polynomial approximation and based on different types of polynomial approximations, we propose (efficient) non-interactive locally differentially private algorithms for learning the set of k-way marginal queries and the set of smooth queries.

Motivation & Objective

• Address the exponential sample complexity in dimension p that plagues prior ERM methods in the non-interactive local differential privacy (LDP) model.
• Overcome the fundamental limitation where general loss functions require sample complexity exponential in p to achieve error α.
• Design efficient protocols with 1-bit communication and O(1) computation per user, while preserving privacy and accuracy.
• Extend the approach to answer k-way marginal queries and smooth queries under LDP using polynomial approximation techniques.
• Provide theoretical guarantees on sample complexity and accuracy for both empirical and population risk minimization.

Proposed method

• Use Bernstein polynomial approximation for (∞,T)-smooth loss functions to reduce sample complexity from exponential to polynomial in α.
• Apply polynomial approximation of the inner product for 1-Lipschitz generalized linear convex loss functions to achieve linear dependence on dimension p.
• Leverage trigonometric polynomial approximation for (h,T)-smooth functions on the hypercube to handle smooth queries.
• Design non-interactive protocols where each user sends only 1 bit of information, using randomized response or similar mechanisms.
• Use moment estimation and coefficient recovery techniques to allow the server to reconstruct query answers from noisy summaries.
• Integrate concentration bounds and error propagation analysis to ensure (α,β)-accuracy with high probability.

Experimental results

Research questions

• RQ1 Can the exponential dependence on dimension p in ERM under non-interactive LDP be removed for certain classes of loss functions?
• RQ2 What conditions on loss functions allow for polynomial sample complexity in the error parameter α while maintaining privacy?
• RQ3 Can efficient 1-bit, O(1)-computation protocols be designed for both users and servers under non-interactive LDP?
• RQ4 Can the polynomial approximation framework be extended to other query classes such as k-way marginals and smooth queries?
• RQ5 What is the trade-off between privacy parameters ε, δ and sample complexity for generalized linear models under LDP?

Key findings

• For (∞,T)-smooth loss functions, the sample complexity for achieving α-error is reduced from α^{-(p+1)} to α^{-c} for some constant c, eliminating exponential dependence on p.
• The proposed algorithm achieves (α,β)-accuracy with sample complexity n = O(ε^{-2} log(1/β) α^{-c}) for some c, under the Bernstein polynomial approximation framework.
• For 1-Lipschitz generalized linear convex loss functions, the sample complexity is linear in p and exponential only in ε and δ, not in p.
• An efficient protocol is designed with 1-bit communication and O(1) computation per user, while maintaining privacy and accuracy.
• For smooth queries in C^h_T, the sample complexity is O(ε^{-2} α^{-(5p+2h)/h} polylog(1/β)) with query answering time Õ((√n ε)^{(4p+4)/(5p+2h) + ...}).
• The framework extends to k-way marginals and smooth queries, providing the first non-interactive LDP algorithms with polynomial sample complexity in α and p.