[Paper Review] Clustered Sampling: Low-Variance and Improved Representativity for Clients Selection in Federated Learning

Yann Fraboni, Yu. Vertogradova+2 more|arXiv (Cornell University)|May 12, 2021

Privacy-Preserving Technologies in Data47 citations

TL;DR

Introduces clustered sampling for selecting clients in federated learning, achieving unbiased aggregation with reduced weight variance and better representativity, improving convergence.

ABSTRACT

This work addresses the problem of optimizing communications between server and clients in federated learning (FL). Current sampling approaches in FL are either biased, or non optimal in terms of server-clients communications and training stability. To overcome this issue, we introduce extit{clustered sampling} for clients selection. We prove that clustered sampling leads to better clients representatitivity and to reduced variance of the clients stochastic aggregation weights in FL. Compatibly with our theory, we provide two different clustering approaches enabling clients aggregation based on 1) sample size, and 2) models similarity. Through a series of experiments in non-iid and unbalanced scenarios, we demonstrate that model aggregation through clustered sampling consistently leads to better training convergence and variability when compared to standard sampling approaches. Our approach does not require any additional operation on the clients side, and can be seamlessly integrated in standard FL implementations. Finally, clustered sampling is compatible with existing methods and technologies for privacy enhancement, and for communication reduction through model compression.

Motivation & Objective

• Motivate the need to optimize server-client communications in federated learning and address biases and high variance in existing client sampling.
• Introduce clustered sampling as an unbiased, low-variance alternative to MD sampling.
• Theoretically establish convergence guarantees for clustered sampling alongside MD sampling as a baseline.
• Show practical clustering-based implementations that improve representativity and reduce aggregation weight variance across iterations.

Proposed method

• Define clustered sampling as a generalization of MD sampling using multiple independent distributions W_k(t) for selecting m clients.
• Prove that satisfying r_{k,i}^t probabilities across m distributions yields unbiased global aggregation (sum r_{k,i}^t = m p_i).
• Prove that clustered sampling retains the same convergence bound as MD sampling (O(1/sqrt(mNT)) + O(mN/T)).
• Provide two concrete clustering approaches: (i) based on client sample size (Algorithm 1), and (ii) based on model/update similarity (Algorithm 2).
• Extend the theory to both uniform unbiasedness and practical implementation without extra client-side operations.
• Discuss compatibility with privacy-preserving and communication-reduction techniques.

Experimental results

Research questions

• RQ1 Does clustered sampling provide unbiased aggregation like MD sampling while reducing the variance of client aggregation weights?
• RQ2 How does clustering influence the representativity of heterogeneous (non-iid, unbalanced) client data in FL?
• RQ3 Can practical clustering-based sampling schemes (by sample size and by similarity) improve convergence speed and stability compared with MD sampling?
• RQ4 What are the theoretical convergence guarantees for clustered sampling relative to existing approaches?
• RQ5 How do the proposed algorithms perform on standard benchmarks (MNIST, CIFAR-10) with non-iid partitions?

Key findings

• Clustered sampling achieves the same convergence bound as MD sampling under suitable conditions, and thus preserves theoretical guarantees for FL optimization.
• Clustered sampling reduces the variance of clients’ aggregation weights compared to MD sampling, improving representativity of each client.
• A higher probability of sampling all clients across iterations is provided by clustering, leading to better representation of heterogenous data.
• Algorithms based on sample size (Algorithm 1) and on similarity (Algorithm 2) are practical, with the latter promoting representation of similar clients.
• Experiments on MNIST and CIFAR-10 show improved training convergence and reduced variability versus MD sampling, especially under higher data heterogeneity.