[Paper Review] FedCM: A Real-time Contribution Measurement Method for Participants in Federated Learning

Boyi Liu, Bingjie Yan+3 more|arXiv (Cornell University)|Sep 8, 2020

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper proposes FedCM, a real-time contribution measurement method for federated learning (FL) participants that uses attention-based aggregation to dynamically assess each agent's impact on model improvement across rounds. FedCM enables timely, lightweight, and sensitive evaluation of data quality and quantity, outperforming state-of-the-art methods like Shapley Value in detecting low-quality or reduced-data agents while being deployable in real-world FL systems.

ABSTRACT

Federated Learning (FL) creates an ecosystem for multiple agents to collaborate on building models with data privacy consideration. The method for contribution measurement of each agent in the FL system is critical for fair credits allocation but few are proposed. In this paper, we develop a real-time contribution measurement method FedCM that is simple but powerful. The method defines the impact of each agent, comprehensively considers the current round and the previous round to obtain the contribution rate of each agent with attention aggregation. Moreover, FedCM updates contribution every round, which enable it to perform in real-time. Real-time is not considered by the existing approaches, but it is critical for FL systems to allocate computing power, communication resources, etc. Compared to the state-of-the-art method, the experimental results show that FedCM is more sensitive to data quantity and data quality under the premise of real-time. Furthermore, we developed federated learning open-source software based on FedCM. The software has been applied to identify COVID-19 based on medical images.

Motivation & Objective

• To address the lack of real-time, fair contribution measurement in federated learning systems that can guide resource allocation and incentive mechanisms.
• To develop a lightweight, computationally efficient method that evaluates agent contributions without offline computation or high overhead.
• To enable accurate detection of agents contributing low-quality or reduced-volume data in real time.
• To support deployable, open-source FL software with integrated contribution measurement for real-world applications like medical image analysis.
• To improve fairness and security in FL by enabling early identification of underperforming or malicious agents through dynamic contribution tracking.

Proposed method

• FedCM uses an attention mechanism to compute dynamic attention weights for each client based on their model updates across training rounds.
• It aggregates contributions by analyzing changes in server-side model parameters before and after each round, using both current and prior round information.
• The method computes contribution rates via a mathematical formulation that combines attention weights and parameter changes across model layers.
• It leverages the attention-based model to assess the impact of each agent’s update on the global model’s performance improvement.
• The system is designed for real-time operation, updating contributions after every communication round, enabling immediate feedback for resource allocation.
• The method is integrated into an open-source FL framework called FedMedical, which supports deployment in privacy-preserving medical AI applications.

Experimental results

Research questions

• RQ1 How can agent contributions in federated learning be measured in real time to support dynamic resource allocation?
• RQ2 To what extent is the proposed method sensitive to variations in data quantity and quality compared to existing offline methods?
• RQ3 Can the method detect and penalize agents providing low-quality or reduced-volume data effectively during training?
• RQ4 How does the attention-based aggregation mechanism improve fairness and responsiveness in contribution measurement compared to Shapley Value?
• RQ5 Can the method be efficiently deployed in real-world FL systems, such as medical image analysis, without significant computational overhead?

Key findings

• FedCM demonstrated higher sensitivity to data quantity and quality than Shapley Value, especially in detecting agents with reduced data volume (e.g., 70% reduction).
• In experiments with random word sequences as data, FedCM successfully identified and assigned low contribution to such agents, similar to Shapley Value but in real time.
• The method showed consistent and stable contribution scores under normal data distribution, with minimal deviation across agents.
• When data volume was reduced by 30% and 70% for different agents, FedCM clearly reflected the relative differences, while Shapley Value failed to capture the distinction effectively.
• The open-source FedMedical software, built on FedCM, successfully enabled privacy-preserving, federated training for COVID-19 detection using medical images across distributed hospitals.
• FedCM enables real-time contribution tracking, making it suitable for dynamic incentive mechanisms and resource allocation in production FL systems.