[Paper Review] Collaborative Learning in the Jungle

El Mahdi El Mhamdi, Sadegh Farhadkhani+4 more|arXiv (Cornell University)|Aug 3, 2020

Privacy-Preserving Technologies in Data29 references4 citations

TL;DR

This paper introduces averaging agreement as a foundational problem for Byzantine collaborative learning, where nodes with heterogeneous data must converge on a vector close to the average of honest nodes' inputs despite up to f nodes behaving arbitrarily. It presents two optimal asynchronous algorithms—one requiring n ≥ 6f+1 with near-optimal averaging constants, and another with optimal resilience at n ≥ 3f+1 using trimmed mean and reliable broadcast—proving equivalence between collaborative learning and averaging agreement and establishing new impossibility theorems in adversarial, heterogeneous settings.

ABSTRACT

We study Byzantine collaborative learning, where $n$ nodes seek to collectively learn from each others' local data. The data distribution may vary from one node to another. No node is trusted, and $f < n$ nodes can behave arbitrarily. We prove that collaborative learning is equivalent to a new form of agreement, which we call averaging agreement. In this problem, nodes start each with an initial vector and seek to approximately agree on a common vector, which is close to the average of honest nodes' initial vectors. We present two asynchronous solutions to averaging agreement, each we prove optimal according to some dimension. The first, based on the minimum-diameter averaging, requires $ n \geq 6f+1$, but achieves asymptotically the best-possible averaging constant up to a multiplicative constant. The second, based on reliable broadcast and coordinate-wise trimmed mean, achieves optimal Byzantine resilience, i.e., $n \geq 3f+1$. Each of these algorithms induces an optimal Byzantine collaborative learning protocol. In particular, our equivalence yields new impossibility theorems on what any collaborative learning algorithm can achieve in adversarial and heterogeneous environments.

Motivation & Objective

• To formalize collaborative learning under Byzantine faults as a new agreement problem: averaging agreement.
• To design asynchronous algorithms that achieve optimal resilience and convergence in heterogeneous, adversarial data environments.
• To prove equivalence between Byzantine collaborative learning and averaging agreement, enabling new impossibility results.
• To establish optimal bounds on convergence accuracy and fault tolerance in distributed learning under arbitrary node behavior.

Proposed method

• Introduces averaging agreement as a new consensus primitive where nodes converge on a vector close to the average of honest nodes' initial vectors.
• Proposes a first algorithm based on minimum-diameter averaging, requiring n ≥ 6f+1, which achieves asymptotically optimal averaging constants.
• Develops a second algorithm using reliable broadcast and coordinate-wise trimmed mean, achieving optimal Byzantine resilience with n ≥ 3f+1.
• Uses the equivalence between collaborative learning and averaging agreement to derive new impossibility theorems for adversarial, heterogeneous learning environments.
• Proves optimality of both algorithms in terms of fault tolerance and convergence accuracy, respectively.
• Applies the two algorithms to construct optimal Byzantine collaborative learning protocols.

Experimental results

Research questions

• RQ1 Can Byzantine collaborative learning be reduced to a new form of agreement, and if so, what are its properties?
• RQ2 What is the minimal number of nodes required to achieve optimal resilience in asynchronous Byzantine collaborative learning?
• RQ3 How can convergence accuracy be maximized under arbitrary node behavior and heterogeneous data distributions?
• RQ4 What are the fundamental limits (impossibility results) of collaborative learning in adversarial, heterogeneous environments?
• RQ5 Can optimal trade-offs between fault tolerance and convergence accuracy be achieved in asynchronous settings?

Key findings

• The paper establishes a formal equivalence between Byzantine collaborative learning and averaging agreement, enabling new theoretical insights into distributed learning under adversarial conditions.
• A protocol based on minimum-diameter averaging achieves asymptotically optimal averaging constants, with the bound n ≥ 6f+1 being necessary for this approach.
• A second protocol using reliable broadcast and coordinate-wise trimmed mean achieves optimal Byzantine resilience with n ≥ 3f+1, matching the theoretical lower bound.
• Both algorithms are proven optimal in their respective dimensions: one in convergence accuracy (averaging constant), the other in fault tolerance.
• The equivalence between collaborative learning and averaging agreement leads to new impossibility theorems, showing inherent limits on what can be achieved in adversarial, heterogeneous learning environments.
• The results demonstrate that optimal Byzantine collaborative learning is achievable through the two proposed asynchronous algorithms, each optimal in a distinct performance metric.