[Paper Review] Benchmarking Algorithms for Federated Domain Generalization

Ruqi Bai, Saurabh Bagchi+1 more|arXiv (Cornell University)|Jul 11, 2023

Domain Adaptation and Few-Shot LearningComputer Science3 citations

TL;DR

This paper introduces the first systematic benchmark methodology for Federated Domain Generalization (FedDG), enabling controlled evaluation of client heterogeneity, dataset diversity, and communication rounds. It evaluates 13 FedDG methods across multiple datasets, revealing significant performance gaps—especially under high client heterogeneity or large client counts—highlighting that FedDG remains unsolved despite progress in centralized DG.

ABSTRACT

While prior domain generalization (DG) benchmarks consider train-test dataset heterogeneity, we evaluate Federated DG which introduces federated learning (FL) specific challenges. Additionally, we explore domain-based heterogeneity in clients' local datasets - a realistic Federated DG scenario. Prior Federated DG evaluations are limited in terms of the number or heterogeneity of clients and dataset diversity. To address this gap, we propose an Federated DG benchmark methodology that enables control of the number and heterogeneity of clients and provides metrics for dataset difficulty. We then apply our methodology to evaluate 14 Federated DG methods, which include centralized DG methods adapted to the FL context, FL methods that handle client heterogeneity, and methods designed specifically for Federated DG. Our results suggest that despite some progress, there remain significant performance gaps in Federated DG particularly when evaluating with a large number of clients, high client heterogeneity, or more realistic datasets. Please check our extendable benchmark code here: https://github.com/inouye-lab/FedDG_Benchmark.

Motivation & Objective

• Address the lack of comprehensive benchmarks for Federated Domain Generalization (FedDG) that account for both client heterogeneity and domain shift.
• Develop a systematic benchmark methodology to control client number, heterogeneity, and dataset difficulty in FedDG settings.
• Evaluate 13 representative FedDG methods under realistic conditions, including large numbers of heterogeneous clients and diverse datasets.
• Identify key challenges in FedDG performance, such as degradation with increasing clients and non-monotonic communication effects.
• Provide recommendations for future evaluations and method design in FedDG to improve robustness and convergence.

Proposed method

• Propose a novel FedDG benchmark methodology that enables controlled splitting of datasets across clients based on domain and heterogeneity levels.
• Introduce a metric to quantify dataset difficulty, enabling systematic comparison across different data distributions.
• Adapt 13 existing methods from three categories: centralized DG methods adapted to FL, FL methods for client heterogeneity, and methods specifically designed for FedDG.
• Implement a communication round control mechanism to study the impact of training frequency on DG performance.
• Use real-world datasets including PACS, IWildCam, Py150, and CivilComments to evaluate under realistic domain and client heterogeneity.
• Apply early stopping and communication scheduling to analyze implicit regularization effects in FedDG.

Experimental results

Research questions

• RQ1 How do FedDG methods perform under increasing numbers of heterogeneous clients, particularly when C ≥ 10?
• RQ2 What is the impact of communication frequency on DG performance in federated settings, and does it follow the in-distribution FL trend?
• RQ3 How does client heterogeneity (controlled via λ) affect convergence and accuracy in FedDG, especially on real-world datasets?
• RQ4 How do different benchmark datasets (e.g., FEMNIST, PACS, IWildCam, Py150) compare in terms of difficulty and client heterogeneity?
• RQ5 To what extent do existing baselines like FedAvg and variance-reduction methods perform in FedDG, and can they serve as strong baselines?

Key findings

• Many FedDG methods degrade significantly when the number of clients exceeds 10, despite performing well on C=3 in original papers, indicating a critical scalability gap.
• Performance does not monotonically improve with more communication rounds; instead, early stopping often yields better DG accuracy, suggesting implicit regularization via limited communication.
• FedAvg and variance-reduction-based FL methods outperform many specialized FedDG methods, especially under high client heterogeneity, indicating strong baseline performance.
• Realistic datasets like IWildCam and Py150 exhibit slower convergence and higher difficulty than PACS or FEMNIST, particularly when client heterogeneity is high (low λ).
• The domain separation case (λ=0), where clients have data from only one domain, remains a major challenge, as centralized DG methods fail and current FedDG methods still underperform.
• There is a strong need for improved convergence rates and better theoretical understanding of how client count and communication frequency affect FedDG performance.