[论文解读] FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout

Samuel Horváth, Stefanos Laskaridis+4 more|arXiv (Cornell University)|Feb 26, 2021

Privacy-Preserving Technologies in Data参考文献 58被引用 64

一句话总结

FjORD 引入 Ordered Dropout 以在异构设备上的联邦学习中实现自适应、嵌套子模型，从而在不对子模型重新训练的情况下提升公平性和准确性。它还包含一种自蒸馏机制，以提升较小子模型的性能。

ABSTRACT

Federated Learning (FL) has been gaining significant traction across different ML tasks, ranging from vision to keyboard predictions. In large-scale deployments, client heterogeneity is a fact and constitutes a primary problem for fairness, training performance and accuracy. Although significant efforts have been made into tackling statistical data heterogeneity, the diversity in the processing capabilities and network bandwidth of clients, termed as system heterogeneity, has remained largely unexplored. Current solutions either disregard a large portion of available devices or set a uniform limit on the model's capacity, restricted by the least capable participants. In this work, we introduce Ordered Dropout, a mechanism that achieves an ordered, nested representation of knowledge in deep neural networks (DNNs) and enables the extraction of lower footprint submodels without the need of retraining. We further show that for linear maps our Ordered Dropout is equivalent to SVD. We employ this technique, along with a self-distillation methodology, in the realm of FL in a framework called FjORD. FjORD alleviates the problem of client system heterogeneity by tailoring the model width to the client's capabilities. Extensive evaluation on both CNNs and RNNs across diverse modalities shows that FjORD consistently leads to significant performance gains over state-of-the-art baselines, while maintaining its nested structure.

研究动机与目标

• Motivate federated learning under strong system heterogeneity where device capabilities vary widely.
• Propose a mechanism to train and deploy nested submodels without retraining for different device tiers.
• Enable dynamic inference-time scaling of model width to match device constraints while preserving knowledge transfer.
• Introduce a self-distillation approach to enhance feature extraction for smaller submodels.

提出的方法

• Introduce Ordered Dropout (OD) to prune model widths in an ordered, nested fashion per layer.
• Train OD-enabled networks in two modes: plain OD and OD with knowledge distillation (OD w/ KD).
• Show that OD recovers SVD in linear mappings, establishing an ordered importance representation.
• Apply FjORD framework on FL by associating p-values with device clusters and performing WA-based aggregation across heterogeneous clients.
• Use a distillation-based loss that combines cross-entropy with KL divergence to transfer knowledge from the largest submodel to smaller submodels.
• Evaluate on CNNs and RNNs across CIFAR10, FEMNIST, and Shakespeare to assess accuracy gains and scalability.]
• research_questions: ["Can Ordered Dropout provide an effective, nested representation that enables variable-width submodels without retraining in federated settings?", "How does FjORD perform in FL with heterogeneous device capabilities compared to state-of-the-art baselines?", "Does knowledge distillation improve the performance of smaller submodels in FjORD?"]
• key_findings:[

实验结果

主要发现

• FjORD consistently outperforms baselines across datasets, with accuracy gains over eFD ranging from 1.53 to 34.87 percentage points on CIFAR10 and 1.57 to 6.27 pp on FEMNIST, while Shakespeare shows smaller gains (0.01 to 0.82 points).
• FjORD+KD yields notable improvements compared to FjORD without KD, especially for mid-to-large submodels (e.g., CIFAR10 and FEMNIST).
• The OD framework enables training a single model that can extract multiple submodels with varying FLOPs and sizes without retraining, and supports dynamic adaptation during inference.
• OD recovers a best-b-rank approximation behavior consistent with SVD in linear mappings, providing theoretical grounding for the ordered importance structure.
• FjORD demonstrates scalability to more device clusters (uniform-5 vs uniform-10) and adaptability to different device distributions (ds=0.5 vs ds=1.0) without significant degradation of smaller submodels.