[Paper Review] FLIS: Clustered Federated Learning via Inference Similarity for Non-IID Data Distribution

Mahdi Morafah, Saeed Vahidian+2 more|arXiv (Cornell University)|Aug 20, 2022

Privacy-Preserving Technologies in Data4 citations

TL;DR

FLIS proposes a novel clustered federated learning framework that groups clients into clusters based on inference similarity of their models, enabling personalized and efficient training under Non-IID data distributions. By leveraging server-side inference similarity without accessing private data, FLIS achieves up to 40% higher accuracy than SOTA global FL and up to 30% higher than SOTA personalized FL, while supporting unseen clients and adaptive clustering without prior knowledge of cluster count.

ABSTRACT

Classical federated learning approaches yield significant performance degradation in the presence of Non-IID data distributions of participants. When the distribution of each local dataset is highly different from the global one, the local objective of each client will be inconsistent with the global optima which incur a drift in the local updates. This phenomenon highly impacts the performance of clients. This is while the primary incentive for clients to participate in federated learning is to obtain better personalized models. To address the above-mentioned issue, we present a new algorithm, FLIS, which groups the clients population in clusters with jointly trainable data distributions by leveraging the inference similarity of clients' models. This framework captures settings where different groups of users have their own objectives (learning tasks) but by aggregating their data with others in the same cluster (same learning task) to perform more efficient and personalized federated learning. We present experimental results to demonstrate the benefits of FLIS over the state-of-the-art benchmarks on CIFAR-100/10, SVHN, and FMNIST datasets. Our code is available at https://github.com/MMorafah/FLIS.

Motivation & Objective

• Address the performance degradation in federated learning caused by Non-IID data distributions, where global models underperform local models.
• Enable personalized model learning for clients with highly skewed or diverse data distributions without sharing private data.
• Develop a clustering mechanism that groups clients based on inference similarity rather than model weights or gradients, allowing flexible and adaptive cluster formation.
• Support dynamic client participation, including unseen clients joining after training, by enabling fast personalization using existing cluster structures.
• Eliminate the need to predefine the number of clusters, making the method robust across varying degrees of data skew.

Proposed method

• FLIS uses inference responses from clients on a small server-held dataset to compute similarity scores between clients, forming a similarity matrix.
• The server applies hard thresholding or hierarchical clustering on the inference similarity matrix to assign clients to clusters without accessing their private data.
• Model aggregation is performed independently within each cluster, allowing each group to train a personalized model tailored to its data distribution.
• Clients in each round select the best-performing cluster model based on local validation loss, enabling personalized adaptation.
• The framework dynamically forms clusters in an online fashion, allowing new clients to join and select the most similar cluster for personalization.
• The method is robust to both highly skewed and slightly skewed Non-IID data, and does not require prior knowledge of the number of clusters.

Experimental results

Research questions

• RQ1 Can inference similarity between client models be used effectively to group clients into meaningful clusters without access to private data?
• RQ2 How does clustering based on inference similarity compare to clustering based on model weights or gradients in terms of personalization and convergence?
• RQ3 Can FLIS achieve superior performance compared to SOTA global and personalized FL methods under Non-IID data settings?
• RQ4 How well does FLIS generalize to unseen clients that join the federation after training has started?
• RQ5 What is the optimal inference similarity threshold β for balancing clustering accuracy and model performance?

Key findings

• FLIS achieves up to 40% higher test accuracy than state-of-the-art global FL baselines (e.g., FedAvg, FedProx) on CIFAR-100 and FMNIST under Non-IID label skew.
• On FMNIST with 20% Non-IID skew, FLIS (DC) achieves 97.51% average local test accuracy, outperforming IFCA (96.29%) and Per-FedAvg (89.88%).
• For CIFAR-10 with 30% Non-IID skew, FLIS converges in just 30 communication rounds, outperforming all SOTA baselines including IFCA and LG.
• FLIS enables unseen clients to achieve high personalization accuracy—97.51% on FMNIST—demonstrating strong generalization to new participants.
• The optimal inference similarity threshold β varies by dataset (e.g., β=0.1 for CIFAR-10, β=0.5 for FMNIST), and performance peaks at these values with minimal clustering error.
• FLIS maintains strong performance across both highly skewed and slightly skewed Non-IID regimes, unlike prior methods that require fixed cluster counts.