[Paper Review] FedEval: A Holistic Evaluation Framework for Federated Learning

Di Chai, Leye Wang+4 more|arXiv (Cornell University)|Nov 19, 2020

Privacy-Preserving Technologies in Data46 references20 citations

TL;DR

This paper proposes FedEval, a holistic evaluation framework for federated learning that standardizes benchmarking through a taxonomy of Privacy, Robustness, Effectiveness, and Efficiency. It introduces FedEval-Core, a structured evaluation model with consistent metrics and settings, and applies it to benchmark seven SOTA FL algorithms, revealing trade-offs and practical deployment guidelines across diverse scenarios.

ABSTRACT

Federated Learning (FL) has been widely accepted as the solution for privacy-preserving machine learning without collecting raw data. While new technologies proposed in the past few years do evolve the FL area, unfortunately, the evaluation results presented in these works fall short in integrity and are hardly comparable because of the inconsistent evaluation metrics and experimental settings. In this paper, we propose a holistic evaluation framework for FL called FedEval, and present a benchmarking study on seven state-of-the-art FL algorithms. Specifically, we first introduce the core evaluation taxonomy model, called FedEval-Core, which covers four essential evaluation aspects for FL: Privacy, Robustness, Effectiveness, and Efficiency, with various well-defined metrics and experimental settings. Based on the FedEval-Core, we further develop an FL evaluation platform with standardized evaluation settings and easy-to-use interfaces. We then provide an in-depth benchmarking study between the seven well-known FL algorithms, including FedSGD, FedAvg, FedProx, FedOpt, FedSTC, SecAgg, and HEAgg. We comprehensively analyze the advantages and disadvantages of these algorithms and further identify the suitable practical scenarios for different algorithms, which is rarely done by prior work. Lastly, we excavate a set of take-away insights and future research directions, which are very helpful for researchers in the FL area.

Motivation & Objective

• To address the lack of standardized, comprehensive evaluation in federated learning due to inconsistent metrics and experimental settings.
• To develop a unified evaluation taxonomy that covers core aspects: Privacy, Robustness, Effectiveness, and Efficiency.
• To build a standardized evaluation platform that enables reproducible and comparable benchmarking of FL algorithms.
• To provide actionable insights on algorithm suitability across different practical scenarios through in-depth comparative analysis.
• To identify research gaps and future directions in FL evaluation, especially regarding privacy attacks and platform-level comparisons.

Proposed method

• Proposes FedEval-Core, a four-dimensional evaluation taxonomy covering Privacy, Robustness, Effectiveness, and Efficiency with well-defined metrics and experimental configurations.
• Designs standardized evaluation protocols for each dimension, including metrics for communication efficiency, model convergence, and privacy leakage.
• Develops an open-source evaluation platform that automates benchmarking with consistent settings and user-friendly interfaces.
• Applies the framework to evaluate seven state-of-the-art FL algorithms: FedSGD, FedAvg, FedProx, FedOpt, FedSTC, SecAgg, and HEAgg.
• Integrates diverse evaluation scenarios, including non-IID data, gradient-based privacy attacks, and communication cost analysis.
• Uses a combination of quantitative metrics and qualitative analysis to compare algorithm trade-offs and identify deployment suitability.

Experimental results

Research questions

• RQ1 How can we standardize the evaluation of federated learning systems to ensure fair and comparable benchmarking across different algorithms?
• RQ2 What are the key trade-offs between privacy, efficiency, robustness, and effectiveness in federated learning algorithms?
• RQ3 Which FL algorithms perform best under specific real-world conditions such as non-IID data or communication constraints?
• RQ4 How do different privacy-preserving techniques (e.g., SecAgg, HEAgg) compare in terms of actual data leakage and performance overhead?
• RQ5 What insights can be drawn from a holistic evaluation that are missed by isolated, single-metric assessments?

Key findings

• FedEval-Core provides a standardized, comprehensive evaluation framework that enables fair and consistent benchmarking across multiple FL algorithms.
• A smaller number of communication rounds does not necessarily imply better efficiency due to increased local computation time, highlighting the need for multi-dimensional efficiency metrics.
• Existing works often neglect the computational cost of local updates, leading to misleading efficiency claims in FL literature.
• Private data leakage from gradients remains a critical concern, and current privacy evaluations using differential privacy or homomorphic encryption are not comparable due to inconsistent assumptions and definitions.
• The benchmarking study reveals distinct strengths and weaknesses of each algorithm: FedAvg excels in convergence speed but struggles with non-IID data; FedProx and FedOpt show better robustness to data heterogeneity; SecAgg and HEAgg provide strong privacy guarantees at the cost of communication overhead.
• The study identifies practical deployment scenarios for each algorithm, offering actionable guidance for researchers and practitioners.