[Paper Review] Federated Active Learning (F-AL): an Efficient Annotation Strategy for Federated Learning

Jin-Hyun Ahn, Kyungsang Kim+2 more|arXiv (Cornell University)|Feb 1, 2022

Privacy-Preserving Technologies in Data32 references4 citations

TL;DR

This paper proposes Federated Active Learning (F-AL), a collaborative active learning strategy within federated learning that reduces annotation costs by enabling clients to jointly select the most informative samples for labeling. F-AL outperforms random sampling and client-level separate active learning (S-AL) by improving global model accuracy with fewer labeled instances through distributed, collaborative uncertainty-based sampling.

ABSTRACT

Federated learning (FL) has been intensively investigated in terms of communication efficiency, privacy, and fairness. However, efficient annotation, which is a pain point in real-world FL applications, is less studied. In this project, we propose to apply active learning (AL) and sampling strategy into the FL framework to reduce the annotation workload. We expect that the AL and FL can improve the performance of each other complementarily. In our proposed federated active learning (F-AL) method, the clients collaboratively implement the AL to obtain the instances which are considered as informative to FL in a distributed optimization manner. We compare the test accuracies of the global FL models using the conventional random sampling strategy, client-level separate AL (S-AL), and the proposed F-AL. We empirically demonstrate that the F-AL outperforms baseline methods in image classification tasks.

Motivation & Objective

• To address the high annotation cost in real-world federated learning (FL) applications, where labeling is often the dominant bottleneck.
• To investigate whether active learning (AL) can be effectively integrated into the FL framework to reduce labeling workload.
• To design a collaborative AL strategy that improves global FL model performance while minimizing per-client labeling effort.
• To compare the performance of F-AL against random sampling and client-level separate AL (S-AL) in FL settings.

Proposed method

• F-AL enables clients to collaboratively perform active learning by jointly identifying the most informative samples for labeling, rather than independently selecting them.
• The method uses uncertainty-based sampling strategies (e.g., uncertainty sampling, core-set, VAAL) in a distributed manner, where clients share model updates and sampling decisions via a central parameter server.
• Clients apply state-of-the-art AL algorithms (e.g., MCDAL, LL, core-set, VAAL) to their local unlabeled data, but the final selection of samples is guided by the global model’s informativeness.
• The global model is updated using federated averaging (FedAvg), and the AL process iteratively selects new samples based on uncertainty or representativeness metrics computed across clients.
• The framework enforces FL constraints by ensuring that sampling decisions are made without centralizing all local data, preserving privacy and communication efficiency.
• F-AL uses a distributed optimization approach to balance local informativeness with global model improvement, avoiding bias toward individual client data distributions.

Experimental results

Research questions

• RQ1 Can active learning be effectively integrated into federated learning to reduce the annotation workload?
• RQ2 How does collaborative active learning (F-AL) compare to random sampling and client-level separate active learning (S-AL) in terms of global model accuracy?
• RQ3 Does F-AL improve the performance of active learning in a distributed FL environment compared to isolated client-level AL?
• RQ4 Why do certain AL strategies (e.g., core-set, VAAL) underperform in FL despite their success in centralized settings?
• RQ5 To what extent does F-AL mitigate the performance degradation caused by limited unlabeled data per client in FL?

Key findings

• F-AL significantly outperforms random sampling and S-AL in test accuracy across multiple image classification benchmarks, including Fashion-MNIST, CIFAR-10, and CIFAR-100.
• The performance gain of F-AL is most pronounced with uncertainty-based AL methods such as uncertainty sampling, MCDAL, and LL, which show substantial improvements over baseline methods.
• F-AL reduces the number of required labeled samples while maintaining or improving global model accuracy, demonstrating effective cost reduction in annotation.
• The core-set and VAAL methods underperform in FL due to insufficient unlabeled data per client, but F-AL improves VAAL’s performance (F-VAAL) through collaborative training of VAE and discriminator.
• F-AL causes a performance drop in independent local learning (IL) because clients prioritize global model improvement over local dataset informativeness, leading to biased sampling across clients.
• Despite local data distribution bias, the aggregate dataset selected by F-AL achieves high global model accuracy, confirming the effectiveness of collaborative sampling in FL.