[Paper Review] Dual Class-Aware Contrastive Federated Semi-Supervised Learning

Qi Guo, Yong Qi+2 more|arXiv (Cornell University)|Nov 16, 2022

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper proposes Dual Class-Aware Contrastive Federated Semi-Supervised Learning (DCCFSSL), a novel federated semi-supervised learning method that reduces model deviation between labeled and unlabeled clients and mitigates confirmation bias via a dual class-aware contrastive module and authentication-reweighted aggregation. DCCFSSL achieves state-of-the-art performance, outperforming standard federated supervised learning even with only 10% labeled clients on CIFAR-10, CIFAR-100, and STL-10.

ABSTRACT

Federated semi-supervised learning (FSSL), facilitates labeled clients and unlabeled clients jointly training a global model without sharing private data. Existing FSSL methods predominantly employ pseudo-labeling and consistency regularization to exploit the knowledge of unlabeled data, achieving notable success in raw data utilization. However, these training processes are hindered by large deviations between uploaded local models of labeled and unlabeled clients, as well as confirmation bias introduced by noisy pseudo-labels, both of which negatively affect the global model's performance. In this paper, we present a novel FSSL method called Dual Class-aware Contrastive Federated Semi-Supervised Learning (DCCFSSL). This method accounts for both the local class-aware distribution of each client's data and the global class-aware distribution of all clients' data within the feature space. By implementing a dual class-aware contrastive module, DCCFSSL establishes a unified training objective for different clients to tackle large deviations and incorporates contrastive information in the feature space to mitigate confirmation bias. Moreover, DCCFSSL introduces an authentication-reweighted aggregation technique to improve the server's aggregation robustness. Our comprehensive experiments show that DCCFSSL outperforms current state-of-the-art methods on three benchmark datasets and surpasses the FedAvg with relabeled unlabeled clients on CIFAR-10, CIFAR-100, and STL-10 datasets. To our knowledge, we are the first to present an FSSL method that utilizes only 10\% labeled clients, while still achieving superior performance compared to standard federated supervised learning, which uses all clients with labeled data.

Motivation & Objective

• Address the large deviation between local models of labeled and unlabeled clients in federated semi-supervised learning (FSSL), which stems from differing training objectives and data heterogeneity.
• Mitigate confirmation bias caused by noisy pseudo-labels in FSSL, which arises from overfitting to self-generated labels during consistency regularization.
• Enable effective collaboration between labeled and unlabeled clients by aligning local and global class-aware distributions in the feature space.
• Improve aggregation robustness in the central server by introducing an authentication-reweighted aggregation mechanism based on reliable samples.
• Demonstrate that FSSL can achieve superior performance even with minimal labeled data—specifically, 10% labeled clients—surpassing fully supervised federated learning on benchmark datasets.

Proposed method

• Introduce a dual class-aware contrastive module that models both local class-aware distributions per client and the global class-aware distribution across all clients in the feature space.
• Formulate a unified contrastive loss that aligns local prototypes with global prototypes, reducing divergence between labeled and unlabeled client models.
• Leverage class prototypes as representative embeddings for each class, enhancing feature space alignment and improving generalization.
• Design an authentication-reweighted aggregation strategy that assigns higher weights to clients with more reliable (authenticated) pseudo-labeled samples, improving server-level model robustness.
• Integrate consistency regularization with pseudo-labeling while using contrastive learning to refine feature representations and reduce noise in pseudo-labels.
• Maintain low communication overhead by adding only a small, fixed-size prototype representation per client, independent of model size.

Experimental results

Research questions

• RQ1 Can a unified training objective be designed to reduce model deviation between labeled and unlabeled clients in federated semi-supervised learning?
• RQ2 How can confirmation bias from noisy pseudo-labels be effectively mitigated in FSSL without relying solely on model confidence thresholds?
• RQ3 To what extent can a dual class-aware contrastive learning mechanism improve alignment between local and global class distributions in the feature space?
• RQ4 How does the performance of DCCFSSL scale with varying ratios of labeled to unlabeled clients, especially when labeled data is extremely limited?
• RQ5 Can DCCFSSL outperform standard federated supervised learning when only 10% of clients are labeled, while maintaining robustness and generalization?

Key findings

• DCCFSSL achieves 88.47% accuracy on CIFAR-10 with 10% labeled clients and 90% unlabeled clients under IID settings, outperforming FedAvg with all labeled clients by 2.92%.
• On CIFAR-100, DCCFSSL achieves 88.36% accuracy with only 10% labeled clients, surpassing standard federated supervised learning by 11.64%.
• Even with just one labeled client, DCCFSSL attains 75.76% accuracy on CIFAR-10 under IID and 54.84% under Non-IID, demonstrating strong generalization with minimal labeled data.
• DCCFSSL improves performance by 34.81% over FedAvg-SL-Lower and 19.26% over FedAvg-SL-Lower in the Non-IID setting when only one client is labeled.
• The method increases communication cost by only 0.1848% to 1.817% across datasets, making it practical for cross-silo federated learning environments.
• DCCFSSL significantly improves training stability and achieves higher AUC and F1 scores across all benchmarks, confirming enhanced robustness and consistency.