[Paper Review] Distributed Learning, Communication Complexity and Privacy

Maria-Florina Balcan, Avrim Blum+2 more|arXiv (Cornell University)|Apr 16, 2012

Privacy-Preserving Technologies in Data22 references113 citations

TL;DR

This paper investigates communication complexity in distributed PAC learning, showing that concepts like teaching dimension and mistake bounds critically affect communication needs. It presents communication-efficient algorithms for classes including decision lists, linear separators, and parity functions—achieving O(d log d) and O(d) bits for decision lists and parity, respectively—while preserving privacy via statistical queries without added communication overhead.

ABSTRACT

We consider the problem of PAC-learning from distributed data and analyze fundamental communication complexity questions involved. We provide general upper and lower bounds on the amount of communication needed to learn well, showing that in addition to VC-dimension and covering number, quantities such as the teaching-dimension and mistake-bound of a class play an important role. We also present tight results for a number of common concept classes including conjunctions, parity functions, and decision lists. For linear separators, we show that for non-concentrated distributions, we can use a version of the Perceptron algorithm to learn with much less communication than the number of updates given by the usual margin bound. We also show how boosting can be performed in a generic manner in the distributed setting to achieve communication with only logarithmic dependence on 1/epsilon for any concept class, and demonstrate how recent work on agnostic learning from class-conditional queries can be used to achieve low communication in agnostic settings as well. We additionally present an analysis of privacy, considering both differential privacy and a notion of distributional privacy that is especially appealing in this context.

Motivation & Objective

• To understand the fundamental communication complexity required for PAC learning when data is distributed across multiple parties.
• To identify key concept class properties—beyond VC-dimension—such as teaching-dimension and mistake-bound that influence communication needs.
• To develop communication-efficient learning algorithms for specific concept classes including conjunctions, decision lists, and linear separators.
• To enable privacy-preserving learning in distributed settings without incurring additional communication costs.
• To demonstrate that boosting and agnostic learning can be performed with logarithmic dependence on 1/ϵ in communication

Proposed method

• Uses statistical query (SQ) frameworks to simulate interactions between entities and their local data, enabling privacy-preserving computation.
• Applies a modified Perceptron algorithm for linear separators that reduces communication rounds to O(√(d log(d/ǫ))/ǫ²) by leveraging non-concentrated distributions.
• Employs distributed boosting with logarithmic dependence on 1/ϵ for any concept class, minimizing total communication.
• Introduces distributional privacy via statistical queries with added Laplace noise to ensure that each entity reveals no more than inherent information from its data distribution.
• Leverages the reliable-useful framework of Rivest and Sloan (1988) to show that non-proper learning of parity functions requires only O(d) bits, unlike proper learning which requires Ω(d²).
• Uses minimal hypothesis exchange: the center computes a minimal hypothesis covering all local hypotheses, ensuring low communication and error bounds

Experimental results

Research questions

• RQ1 What is the fundamental communication complexity of learning a concept class in a distributed setting?
• RQ2 How do concept class properties such as teaching-dimension and mistake-bound affect communication requirements?
• RQ3 Can boosting be performed in a distributed setting with communication scaling logarithmically in 1/ϵ?
• RQ4 Can privacy be preserved in distributed learning without increasing communication cost?
• RQ5 What is the communication cost for learning specific classes like decision lists, parity functions, and linear separators under non-concentrated distributions?

Key findings

• For decision lists over {0,1}^d, communication complexity is Θ(d log d) bits, which is tight.
• For parity functions, non-proper learning requires only O(d) bits of communication, a significant improvement over the Ω(d²) lower bound for proper learning.
• For linear separators under non-concentrated distributions, communication can be reduced to O(√(d log(d/ǫ))/ǫ²) rounds, each sending a single hypothesis vector, far below the O(d/ǫ²) updates of standard Perceptron.
• Distributed boosting achieves error O(opt(H)) + ǫ with communication depending only logarithmically on 1/ϵ for any concept class.
• Agnostic learning can be performed with O(log(1/ϵ)) communication dependence on 1/ϵ by adapting Balcan and Hanneke (2012) results.
• Distributional privacy can be achieved with sample size O(M² log³(M/δ)/(α²τ²)) for M queries, preserving privacy without communication overhead when using statistical queries