[Paper Review] Federated Semi-Supervised Learning with Inter-Client Consistency & Disjoint Learning

Wonyong Jeong, Jaehong Yoon+2 more|arXiv (Cornell University)|Jun 22, 2020

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper proposes Federated Matching (FedMatch), a novel framework for Federated Semi-Supervised Learning (FSSL) that addresses two realistic scenarios: labeled data at clients (labels-at-client) and labeled data only at the server (labels-at-server). FedMatch enforces inter-client consistency via a novel consistency loss and uses parameter decomposition to separate supervised and unsupervised learning, reducing interference and communication costs. It significantly outperforms both local SSL and naive FL+SSL combinations on non-i.i.d. and i.i.d. data across multiple benchmarks.

ABSTRACT

While existing federated learning approaches mostly require that clients have fully-labeled data to train on, in realistic settings, data obtained at the client-side often comes without any accompanying labels. Such deficiency of labels may result from either high labeling cost, or difficulty of annotation due to the requirement of expert knowledge. Thus the private data at each client may be either partly labeled, or completely unlabeled with labeled data being available only at the server, which leads us to a new practical federated learning problem, namely Federated Semi-Supervised Learning (FSSL). In this work, we study two essential scenarios of FSSL based on the location of the labeled data. The first scenario considers a conventional case where clients have both labeled and unlabeled data (labels-at-client), and the second scenario considers a more challenging case, where the labeled data is only available at the server (labels-at-server). We then propose a novel method to tackle the problems, which we refer to as Federated Matching (FedMatch). FedMatch improves upon naive combinations of federated learning and semi-supervised learning approaches with a new inter-client consistency loss and decomposition of the parameters for disjoint learning on labeled and unlabeled data. Through extensive experimental validation of our method in the two different scenarios, we show that our method outperforms both local semi-supervised learning and baselines which naively combine federated learning with semi-supervised learning. The code is available at https://github.com/wyjeong/FedMatch.

Motivation & Objective

• To address the practical challenge of limited or absent labels in federated learning, where clients often have only partially labeled or completely unlabeled data.
• To propose a unified framework for Federated Semi-Supervised Learning (FSSL) that handles two realistic scenarios: labels-at-client and labels-at-server.
• To reduce interference between supervised and unsupervised learning by decomposing model parameters into dense (labeled) and sparse (unlabeled) components.
• To minimize communication costs by transmitting only parameter differences across rounds, leveraging the decomposition.
• To improve model generalization through inter-client consistency loss, ensuring agreement across clients' predictions.

Proposed method

• Introduces an inter-client consistency loss that maximizes agreement between model predictions across different clients, promoting knowledge distillation across the federated network.
• Employs a parameter decomposition strategy that splits model weights into two parts: a dense component for labeled data and a sparse additive component for unlabeled data.
• Uses the sparse additive parameter update to isolate unsupervised learning from supervised learning, reducing catastrophic forgetting and interference.
• Reduces communication cost by transmitting only the difference between global and local model parameters, rather than full model weights.
• Applies standard semi-supervised learning techniques (e.g., pseudo-labeling, data augmentation) within the client’s local training, using the decomposed parameters for disjoint learning.
• Aggregates local models at the server using FedAvg-style averaging, but with consistency regularization enforced across clients.

Experimental results

Research questions

• RQ1 Can a federated learning framework effectively leverage both labeled and unlabeled data when labels are only available at the server?
• RQ2 How can inter-client consistency be enforced in a decentralized setting without sharing raw data or labels?
• RQ3 Can parameter decomposition reduce interference between supervised and unsupervised learning in federated semi-supervised learning?
• RQ4 Does FedMatch outperform naive combinations of federated learning and semi-supervised learning in both non-i.i.d. and i.i.d. data settings?
• RQ5 Can communication efficiency be improved through parameter difference transmission in a semi-supervised federated setting?

Key findings

• FedMatch significantly outperforms local semi-supervised learning and naive combinations of FL and SSL on both labels-at-client and labels-at-server scenarios.
• On CIFAR-10 with 20% labeled data, FedMatch achieves 92.1% test accuracy in the labels-at-client setting, outperforming the baseline FedProx-UDA/FixMatch.
• In the more challenging labels-at-server scenario, FedMatch achieves 89.3% accuracy with 20% labeled data, demonstrating effectiveness even when clients have no labels.
• Performance improves consistently with increasing labeled data, while baseline models (e.g., FedProx-UDA/FixMatch) show degradation when label count increases from 5 to 10.
• The method reduces communication cost by transmitting only parameter differences, enabling efficient model updates without full model transfer.
• The inter-client consistency loss leads to more robust and aligned predictions across clients, improving generalization in non-i.i.d. data settings.