[Paper Review] Locally Private Learning without Interaction Requires Separation

Amit Daniely, Vitaly Feldman|arXiv (Cornell University)|Sep 24, 2018

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper establishes that non-interactive locally private learning of distribution-independent PAC classes requires exponential sample complexity unless the class has polynomially small margin complexity. It proves that linear separators and decision lists—naturally occurring classes with low margin complexity—still require exponentially many samples non-interactively, establishing the first exponential separation between interactive and non-interactive LDP learning for natural problems.

ABSTRACT

We consider learning under the constraint of local differential privacy (LDP). For many learning problems known efficient algorithms in this model require many rounds of communication between the server and the clients holding the data points. Yet multi-round protocols are prohibitively slow in practice due to network latency and, as a result, currently deployed large-scale systems are limited to a single round. Despite significant research interest, very little is known about which learning problems can be solved by such non-interactive systems. The only lower bound we are aware of is for PAC learning an artificial class of functions with respect to a uniform distribution (Kasiviswanathan et al. 2011). We show that the margin complexity of a class of Boolean functions is a lower bound on the complexity of any non-interactive LDP algorithm for distribution-independent PAC learning of the class. In particular, the classes of linear separators and decision lists require exponential number of samples to learn non-interactively even though they can be learned in polynomial time by an interactive LDP algorithm. This gives the first example of a natural problem that is significantly harder to solve without interaction and also resolves an open problem of Kasiviswanathan et al. (2011). We complement this lower bound with a new efficient learning algorithm whose complexity is polynomial in the margin complexity of the class. Our algorithm is non-interactive on labeled samples but still needs interactive access to unlabeled samples. All of our results also apply to the statistical query model and any model in which the number of bits communicated about each data point is constrained.

Motivation & Objective

• To identify the fundamental limitations of non-interactive locally private learning (LDP) for distribution-independent PAC learning.
• To close the gap left by prior work, which only showed exponential lower bounds for artificial, non-natural function classes under uniform distributions.
• To establish a tight connection between margin complexity and the sample complexity of non-interactive LDP learning.
• To resolve an open problem posed by [KLNRS11] by showing that natural classes like linear separators and decision lists are exponentially harder to learn non-interactively.
• To extend the lower bound to related models such as the statistical query (SQ) model and bounded-communication protocols.

Proposed method

• Introduces margin complexity as a key complexity measure for Boolean function classes, defined as the inverse of the largest margin achievable via a linear embedding.
• Proves that any non-interactive LDP algorithm for distribution-independent PAC learning must require sample complexity exponential in the margin complexity of the class.
• Uses a reduction from the statistical query (SQ) model, showing that non-interactive LDP algorithms are simulatable by non-adaptive SQ algorithms.
• Applies known SQ lower bounds to the margin complexity measure, leveraging results from communication complexity and learning theory.
• Constructs a new non-interactive LDP learning algorithm whose sample complexity is polynomial in the margin complexity, showing tightness of the bound.
• Extends the results to the bounded-communication model, showing that the same lower bound applies when each sample is communicated using only a limited number of bits.

Experimental results

Research questions

• RQ1 Can non-interactive LDP algorithms efficiently learn natural classes such as linear separators and decision lists in the distribution-independent PAC setting?
• RQ2 Is margin complexity a fundamental barrier to non-interactive learning in LDP and related models?
• RQ3 Does the absence of interaction lead to an exponential sample complexity gap for natural learning problems in the LDP model?
• RQ4 Can the margin complexity of a class be used to characterize the sample complexity of non-interactive learning in LDP and SQ models?
• RQ5 What is the trade-off between the number of rounds and sample complexity in LDP learning, particularly for classes like halfspaces?

Key findings

• The margin complexity of a class of Boolean functions is a lower bound on the sample complexity of any non-interactive LDP algorithm for distribution-independent PAC learning.
• Linear separators and decision lists, which have polynomially bounded margin complexity, require exponentially many samples to learn non-interactively in the LDP model.
• This establishes the first exponential separation between the power of interactive and non-interactive LDP protocols for a natural learning problem.
• The lower bound extends to the statistical query (SQ) model and to the bounded-communication model, where each data point is communicated using at most ℓ bits.
• A new non-interactive LDP learning algorithm is constructed whose sample complexity is polynomial in the margin complexity, showing the bound is tight.
• The results imply that any label-non-adaptive ℓ-communication-bounded algorithm for PAC learning a class C must use at least Ω(MC(C)^{2/3}/2^ℓ) samples, where MC(C) is the margin complexity of C.