[Paper Review] Trusted AI in Multi-agent Systems: An Overview of Privacy and Security for Distributed Learning

Chuan Ma, Jun Li+6 more|arXiv (Cornell University)|Feb 18, 2022

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper proposes a four-level framework—pre-processed data, learning models, extracted knowledge, and intermediate results—to analyze privacy and security risks in distributed machine learning within multi-agent systems. It surveys state-of-the-art threats and defenses across these levels, emphasizing federated learning and Trusted AI, and outlines future research directions for secure, privacy-preserving distributed AI.

ABSTRACT

Motivated by the advancing computational capacity of distributed end-user equipments (UEs), as well as the increasing concerns about sharing private data, there has been considerable recent interest in machine learning (ML) and artificial intelligence (AI) that can be processed on on distributed UEs. Specifically, in this paradigm, parts of an ML process are outsourced to multiple distributed UEs, and then the processed ML information is aggregated on a certain level at a central server, which turns a centralized ML process into a distributed one, and brings about significant benefits. However, this new distributed ML paradigm raises new risks of privacy and security issues. In this paper, we provide a survey of the emerging security and privacy risks of distributed ML from a unique perspective of information exchange levels, which are defined according to the key steps of an ML process, i.e.: i) the level of preprocessed data, ii) the level of learning models, iii) the level of extracted knowledge and, iv) the level of intermediate results. We explore and analyze the potential of threats for each information exchange level based on an overview of the current state-of-the-art attack mechanisms, and then discuss the possible defense methods against such threats. Finally, we complete the survey by providing an outlook on the challenges and possible directions for future research in this critical area.

Motivation & Objective

• To address growing concerns about privacy and security in distributed machine learning (DML) due to increasing data sharing across decentralized edge devices.
• To identify and categorize emerging threats in DML by analyzing information exchange at four distinct levels: pre-processed data, models, knowledge, and intermediate results.
• To survey state-of-the-art attack mechanisms and defense techniques tailored to each information exchange level in multi-agent learning systems.
• To examine the role of standards, regulations (e.g., GDPR, HIPAA, IEEE), and ethical frameworks in enabling trusted AI in distributed learning environments.
• To outline open challenges and future research directions for building secure, private, and trustworthy distributed AI systems.

Proposed method

• Proposes a novel four-level abstraction framework to classify privacy and security risks in distributed ML based on the stage of information exchange: pre-processed data, trained models, extracted knowledge, and intermediate computation results.
• Analyzes threat models at each level, including membership inference, model inversion, model stealing, and data poisoning attacks, with a focus on federated learning and multi-agent collaboration.
• Reviews defense mechanisms such as differential privacy, secure aggregation, homomorphic encryption, and adversarial training, evaluating their applicability per information exchange level.
• Evaluates the role of IEEE standards (e.g., IEEE 3652.1-2020, IEEE P2089, IEEE 1363) and regulatory frameworks (e.g., GDPR, HIPAA, NIST) in enabling secure and privacy-preserving distributed learning.
• Integrates ethical system design principles via the IEEE P7000 series to guide responsible development of distributed AI systems.
• Synthesizes findings into a comprehensive survey with a focus on practical deployment challenges and research gaps in Trusted AI for multi-agent systems.

Experimental results

Research questions

• RQ1 How do privacy and security threats vary across different stages of information exchange in distributed machine learning?
• RQ2 What are the most critical attack vectors at each of the four abstraction levels (data, model, knowledge, intermediate results) in multi-agent learning systems?
• RQ3 Which defense mechanisms are most effective at mitigating threats at each information exchange level, and what are their trade-offs?
• RQ4 How do existing standards and regulations (e.g., IEEE, GDPR, HIPAA) support or constrain the development of trusted distributed AI systems?
• RQ5 What are the key open challenges and future research directions for achieving robust privacy and security in federated and distributed learning?

Key findings

• The four-level framework—pre-processed data, learning models, extracted knowledge, and intermediate results—provides a systematic way to classify and analyze privacy and security risks in distributed ML.
• Threats such as membership inference and model inversion are most severe at the model and intermediate result levels, where model parameters and gradients can leak sensitive information.
• Differential privacy and secure aggregation are effective defenses at the model level, reducing the risk of model inversion and membership inference attacks.
• IEEE 3652.1-2020 provides a standardized architectural framework for federated learning that ensures privacy, security, and regulatory compliance across distributed systems.
• Regulatory frameworks like GDPR and HIPAA impose strict requirements on data processing and consent, directly influencing system design and threat mitigation strategies.
• Despite advances, significant challenges remain in balancing model utility, efficiency, and privacy, especially in real-time, large-scale multi-agent environments.