[Paper Review] Re-thinking Federated Active Learning based on Inter-class Diversity

SangMook Kim, Sangmin Bae+2 more|arXiv (Cornell University)|Mar 22, 2023

Data Quality and ManagementDecision Sciences3 citations

TL;DR

This paper proposes LoGo, a novel federated active learning (FAL) sampling strategy that integrates both local-only and global query selectors to address inter-class imbalance in heterogeneous FL settings. By using a two-step cluster-based selection—first ensuring local diversity via local-only models and then refining queries with global model awareness—LoGo achieves robust performance across diverse data distributions, consistently outperforming six baselines in 38 experimental settings.

ABSTRACT

Although federated learning has made awe-inspiring advances, most studies have assumed that the client's data are fully labeled. However, in a real-world scenario, every client may have a significant amount of unlabeled instances. Among the various approaches to utilizing unlabeled data, a federated active learning framework has emerged as a promising solution. In the decentralized setting, there are two types of available query selector models, namely 'global' and 'local-only' models, but little literature discusses their performance dominance and its causes. In this work, we first demonstrate that the superiority of two selector models depends on the global and local inter-class diversity. Furthermore, we observe that the global and local-only models are the keys to resolving the imbalance of each side. Based on our findings, we propose LoGo, a FAL sampling strategy robust to varying local heterogeneity levels and global imbalance ratio, that integrates both models by two steps of active selection scheme. LoGo consistently outperforms six active learning strategies in the total number of 38 experimental settings.

Motivation & Objective

• Address the challenge of class imbalance in federated active learning (FAL), where both local client data and global model data may exhibit significant inter-class skew.
• Investigate the performance dominance of two query selector types—global and local-only models—under varying levels of local data heterogeneity and global class imbalance.
• Design a robust FAL sampling strategy that adapts to unknown or varying degrees of local and global data imbalance without prior knowledge.
• Develop a method that leverages both local and global model insights to improve query selection efficiency and model generalization in decentralized learning.

Proposed method

• Propose a two-stage cluster-based active learning framework: a macro step using k-means clustering on hallucinated gradient space from local-only models to ensure local inter-class diversity.
• Perform a micro step using one EM iteration: E-step assigns instances to clusters from the macro step, and M-step samples from each cluster using the global model to prioritize global minority classes.
• Integrate both local and global model knowledge implicitly: local-only models capture client-specific data distribution, while the global model captures global data imbalance.
• Use a cluster-wise sampling strategy that conservatively preserves local diversity while incorporating global minority class awareness through the global model.
• Leverage the Dirichlet distribution concentration parameter (α) as a proxy for local heterogeneity and the global imbalance ratio (ρ) to analyze selector dominance.
• Design a data-agnostic strategy that does not require prior knowledge of α or ρ, making it robust across diverse real-world FL scenarios.

Experimental results

Research questions

• RQ1 How does the performance of global versus local-only query selectors in FAL vary with local heterogeneity (α) and global imbalance ratio (ρ)?
• RQ2 Can a unified active learning strategy be designed that effectively combines the strengths of both global and local-only query selectors across diverse data distribution regimes?
• RQ3 To what extent does integrating local and global model insights improve query selection quality and downstream model accuracy in federated active learning?
• RQ4 Does a two-step cluster-based sampling strategy outperform naive ensemble methods that average or rank predictions from global and local models?
• RQ5 How robust is the proposed LoGo method across diverse datasets and varying labeling budgets in terms of test accuracy and convergence speed?

Key findings

• The superiority of global versus local-only query selectors in FAL is not fixed but depends on the interplay between local heterogeneity (α) and global imbalance ratio (ρ).
• When local heterogeneity is high (low α), local-only models outperform global models because they better detect informative instances for minority classes on each client.
• When global imbalance is high (high ρ), global models dominate because they capture the broader data distribution and can prioritize underrepresented global classes.
• LoGo consistently achieves the highest test accuracy across 38 experimental settings, outperforming six baselines including Entropy, BADGE, GCNAL, and ALFA-Mix on CIFAR-10, SVHN, PathMNIST, and DermaMNIST.
• In Table 3 (α=0.1), LoGo achieves 86.02% accuracy on CIFAR-10 at 80% labeling budget—surpassing the next best baseline (GCNAL at 85.10%)—and 86.02% on SVHN, again exceeding all others.
• LoGo significantly outperforms three naive ensemble methods (logit averaging, rank-based weighting, fine-tuning) in Table 4, demonstrating that simple fusion of models is insufficient compared to the structured two-step cluster-based integration in LoGo.