[Paper Review] Byzantine Resilient Distributed Multi-Task Learning

Jiani Li, Waseem Abbas+1 more|arXiv (Cornell University)|Oct 25, 2020

Domain Adaptation and Few-Shot Learning44 references4 citations

TL;DR

This paper proposes a Byzantine-resilient online weight adjustment rule for distributed multi-task learning that uses accumulated loss to measure similarity between agents' models. By filtering out neighbors with high losses, the method ensures normal agents converge to the global minimum despite an arbitrary number of Byzantine agents, achieving resilience without requiring prior knowledge of the number of faulty agents and outperforming non-cooperative learning in both convex and non-convex settings.

ABSTRACT

Distributed multi-task learning provides significant advantages in multi-agent networks with heterogeneous data sources where agents aim to learn distinct but correlated models simultaneously.However, distributed algorithms for learning relatedness among tasks are not resilient in the presence of Byzantine agents. In this paper, we present an approach for Byzantine resilient distributed multi-task learning. We propose an efficient online weight assignment rule by measuring the accumulated loss using an agent's data and its neighbors' models. A small accumulated loss indicates a large similarity between the two tasks. In order to ensure the Byzantine resilience of the aggregation at a normal agent, we introduce a step for filtering out larger losses. We analyze the approach for convex models and show that normal agents converge resiliently towards the global minimum.Further, aggregation with the proposed weight assignment rule always results in an improved expected regret than the non-cooperative case. Finally, we demonstrate the approach using three case studies, including regression and classification problems, and show that our method exhibits good empirical performance for non-convex models, such as convolutional neural networks.

Motivation & Objective

• Address the vulnerability of existing distributed multi-task learning methods to Byzantine agents that send malicious updates.
• Develop a computationally efficient, online method to estimate task similarities without centralized coordination.
• Ensure convergence resilience for normal agents even when all neighbors are Byzantine, without requiring a predefined bound on the number of faulty agents.
• Improve expected regret over non-cooperative learning through adaptive, loss-based neighbor weighting.
• Demonstrate effectiveness in both convex and non-convex models, including deep neural networks, across diverse real-world case studies.

Proposed method

• Measure similarity between agents using the accumulated loss of each agent’s data under its neighbors’ models, where lower loss indicates higher similarity.
• Introduce a filtering step that excludes neighbors whose loss exceeds the agent’s own loss, thereby mitigating influence from Byzantine agents.
• Assign dynamic weights to neighbors based on their normalized loss values, ensuring robust aggregation in the presence of adversarial behavior.
• Use a linear-time computation strategy for weight assignment, scalable to large neighborhoods and high-dimensional data.
• Integrate the loss-based weighting rule into a distributed multi-task learning framework with iterative model exchange and aggregation.
• Prove theoretical convergence to the global minimum under convexity assumptions and show improved expected regret compared to non-cooperative learning.

Experimental results

Research questions

• RQ1 Can a distributed multi-task learning system remain resilient to Byzantine agents when using model parameter distance as a similarity measure?
• RQ2 How can task similarity be estimated online in a decentralized, scalable, and Byzantine-resilient manner?
• RQ3 Does aggregation using loss-based weights improve expected regret compared to non-cooperative learning in the presence of adversarial agents?
• RQ4 Can the proposed method ensure convergence to the global minimum even when all neighbors are Byzantine?
• RQ5 How does the method perform empirically on non-convex models such as convolutional neural networks in real-world applications?

Key findings

• The proposed loss-based weight assignment rule consistently outperforms baseline rules and non-cooperative learning in terms of average loss and classification accuracy across all case studies.
• In the extreme case with only one normal agent and all others Byzantine, the method achieves performance identical to non-cooperative learning, proving resilience to an arbitrary number of Byzantine agents.
• The method ensures that normal agents converge resiliently toward the global minimum under convex models, even when all neighbors are adversarial.
• Aggregation using the proposed weights results in strictly improved expected regret compared to non-cooperative learning, as theoretically proven.
• Empirical results on regression, human action recognition, and digit classification show strong performance on non-convex models, including convolutional neural networks.
• The filtering mechanism effectively suppresses malicious updates by identifying and discarding neighbors with abnormally high losses, enhancing robustness.