[Paper Review] Benchmarking Semi-supervised Federated Learning

Zhang Zheng-ming, Zhewei Yao+4 more|arXiv (Cornell University)|Aug 26, 2020

Privacy-Preserving Technologies in Data39 references28 citations

TL;DR

This paper proposes a semi-supervised federated learning framework where edge devices hold only unlabeled data and a central server holds limited labeled data, addressing non-iid class distributions. It introduces a consistency loss with group normalization and a novel grouping-based model averaging method, achieving up to 10.79% accuracy gain on EMNIST compared to non-grouping baselines.

ABSTRACT

Federated learning promises to use the computational power of edge devices while maintaining user data privacy. Current frameworks, however, typically make the unrealistic assumption that the data stored on user devices come with ground truth labels, while the server has no data. In this work, we consider the more realistic scenario where the users have only unlabeled data and the server has a limited amount of labeled data. In this semi-supervised federated learning (ssfl) setting, the data distribution can be non-iid, in the sense of different distributions of classes at different users. We define a metric, $R$, to measure this non-iidness in class distributions. In this setting, we provide a thorough study on different factors that can affect the final test accuracy, including algorithm design (such as training objective), the non-iidness $R$, the communication period $T$, the number of users $K$, the amount of labeled data in the server $N_s$, and the number of users $C_k\leq K$ that communicate with the server in each communication round. We evaluate our ssfl framework on Cifar-10, SVHN, and EMNIST. Overall, we find that a simple consistency loss-based method, along with group normalization, achieves better generalization performance, even compared to previous supervised federated learning settings. Furthermore, we propose a novel grouping-based model average method to improve convergence efficiency, and we show that this can boost performance by up to 10.79% on EMNIST, compared to the non-grouping based method.

Motivation & Objective

• To address the realistic scenario in federated learning where clients have only unlabeled data and the server holds a small amount of labeled data.
• To study the impact of various factors—such as non-iid class distribution, communication period, and number of clients—on model performance in semi-supervised federated learning.
• To evaluate and compare different algorithmic designs, including training objectives and aggregation strategies, under realistic data and communication constraints.
• To propose and validate a novel grouping-based model averaging technique to improve convergence and test accuracy.

Proposed method

• Introduces a metric $ R $ to quantify non-iidness in class distributions across clients.
• Employs a consistency loss-based training objective to leverage unlabeled client data effectively.
• Applies group normalization to improve generalization in the semi-supervised setting.
• Proposes a grouping-based model averaging strategy that aggregates models from selected client groups rather than all clients in each round.
• Uses Cifar-10, SVHN, and EMNIST datasets to evaluate the framework under varying levels of data non-iidness and label scarcity.
• Employs a communication period $ T $, number of clients $ K $, and number of communicating clients $ C_k \leq K $ as configurable system parameters.

Experimental results

Research questions

• RQ1 How does the level of non-iidness in class distributions ($ R $) affect model accuracy in semi-supervised federated learning?
• RQ2 What is the impact of communication period $ T $, number of clients $ K $, and number of communicating clients $ C_k $ on model convergence and performance?
• RQ3 How does the proposed grouping-based model averaging method compare to standard averaging in terms of accuracy and convergence efficiency?
• RQ4 Can consistency loss with group normalization outperform traditional supervised federated learning methods in this semi-supervised setup?

Key findings

• A simple consistency loss-based method with group normalization achieves better generalization than previous supervised federated learning methods, even under semi-supervised conditions.
• The proposed grouping-based model averaging method improves test accuracy by up to 10.79% on the EMNIST dataset compared to non-grouping baselines.
• The level of non-iidness in class distributions, measured by $ R $, significantly influences model performance, with higher $ R $ generally degrading accuracy.
• The number of communicating clients $ C_k $ and communication period $ T $ have measurable effects on convergence speed and final model accuracy.
• The amount of labeled data on the server ($ N_s $) plays a critical role, with performance improving as $ N_s $ increases, though gains diminish beyond a threshold.