[Paper Review] Learn Electronic Health Records by Fully Decentralized Federated Learning

Songtao Lu, Yawen Zhang+2 more|arXiv (Cornell University)|Dec 4, 2019

Privacy-Preserving Technologies in Data21 references20 citations

TL;DR

This paper proposes a fully decentralized federated learning framework using decentralized stochastic gradient tracking (DSGT) to train machine learning models on distributed electronic health records (EHRs) while preserving data privacy and improving communication efficiency. By performing local updates and enabling peer-to-peer communication among hospitals, the method achieves linear speedup and converges to optimal solutions with significantly fewer communication rounds than centralized or star-networked approaches.

ABSTRACT

Federated learning opens a number of research opportunities due to its high communication efficiency in distributed training problems within a star network. In this paper, we focus on improving the communication efficiency for fully decentralized federated learning over a graph, where the algorithm performs local updates for several iterations and then enables communications among the nodes. In such a way, the communication rounds of exchanging the common interest of parameters can be saved significantly without loss of optimality of the solutions. Multiple numerical simulations based on large, real-world electronic health record databases showcase the superiority of the decentralized federated learning compared with classic methods.

Motivation & Objective

• To address the challenge of training high-quality machine learning models on decentralized, non-identical electronic health records (EHRs) without compromising patient privacy.
• To improve communication efficiency in federated learning by eliminating the need for a central parameter server, especially in real-world healthcare settings where data sharing is restricted by regulations like HIPAA.
• To develop a decentralized optimization framework that enables consensus convergence across hospitals while handling data heterogeneity inherent in real-world EHR datasets.
• To theoretically and empirically demonstrate that decentralized federated learning with DSGT achieves linear speedup and faster convergence with fewer communication rounds compared to classical federated learning methods.

Proposed method

• The method employs a fully decentralized network of 20 hospitals (nodes), where each maintains its own EHR data and communicates only with neighboring nodes via a graph-structured topology.
• It integrates local stochastic gradient updates with decentralized gradient tracking (DSGT), which maintains estimates of the global gradient to reduce error from data heterogeneity.
• The algorithm performs Q local updates per communication round, reducing the number of communication rounds required for convergence.
• The core update rule uses a step size of α^r ~ O(√(N/r)) and leverages a doubly stochastic weight matrix W to ensure consensus across nodes.
• The approach is theoretically grounded in convergence analysis showing an optimality gap of O(σ²/(N√T)) for large T, indicating linear speedup with respect to the number of nodes.
• Numerical experiments compare DSGT-based federated learning (FD-DSGT) with DSGD, DSGT, and federated DSGD, using a shallow neural network with 42-dimensional features on a real Alzheimer’s and MCI EHR dataset.

Experimental results

Research questions

• RQ1 Can a fully decentralized federated learning framework achieve comparable model performance to centralized training while preserving data privacy in real-world EHR systems?
• RQ2 How does the integration of local updates with decentralized gradient tracking (DSGT) improve communication efficiency and convergence speed in non-identical, heterogeneous EHR data?
• RQ3 What is the theoretical convergence behavior of DSGT-based federated learning in a decentralized setting with non-i.i.d. data, and does it achieve linear speedup with respect to the number of nodes?
• RQ4 How does the proposed method compare to classical federated learning and decentralized optimization baselines in terms of communication rounds and optimality gap?

Key findings

• The proposed fully decentralized federated learning with DSGT achieves a convergence rate of O(σ²/(N√T)), demonstrating linear speedup with respect to the number of nodes, which is the first theoretical result of its kind for DSGT in non-convex settings.
• Numerical results show that FD-DSGT converges significantly faster than classical DSGD and DSGT in terms of communication rounds, achieving a smaller optimality gap.
• The method reduces communication rounds substantially compared to standard federated learning, as shown in Figure 2, where FD-DSGT reaches lower optimality gaps with fewer exchanges.
• The t-SNE visualization in Figure 1 confirms data heterogeneity across hospitals, validating the need for decentralized methods that can handle non-identical data distributions.
• The algorithm maintains consensus across nodes, with the average iterate error ‖θ_i^r - θ̄^r‖² shrinking over time, indicating effective information sharing without a central fusion center.