[Paper Review] A Communication Efficient Collaborative Learning Framework for Distributed Features

Yang Liu, Yan Kang+6 more|arXiv (Cornell University)|Dec 24, 2019

Privacy-Preserving Technologies in Data37 references57 citations

TL;DR

The paper proposes FedBCD, a communication-efficient framework for collaboratively training models with vertically partitioned features, achieving O(1/√T) accuracy with O(√T) communication rounds by performing multiple local updates before synchronization.

ABSTRACT

We introduce a collaborative learning framework allowing multiple parties having different sets of attributes about the same user to jointly build models without exposing their raw data or model parameters. In particular, we propose a Federated Stochastic Block Coordinate Descent (FedBCD) algorithm, in which each party conducts multiple local updates before each communication to effectively reduce the number of communication rounds among parties, a principal bottleneck for collaborative learning problems. We analyze theoretically the impact of the number of local updates and show that when the batch size, sample size, and the local iterations are selected appropriately, within $T$ iterations, the algorithm performs $\mathcal{O}(\sqrt{T})$ communication rounds and achieves some $\mathcal{O}(1/\sqrt{T})$ accuracy (measured by the average of the gradient norm squared). The approach is supported by our empirical evaluations on a variety of tasks and datasets, demonstrating advantages over stochastic gradient descent (SGD) approaches.

Motivation & Objective

• Motivate learning across data silos where parties hold different features for the same users without sharing raw data or models.
• Develop a communication-efficient algorithm for vertically partitioned data that minimizes inter-party communication.
• Provide theoretical convergence guarantees for the proposed FedBCD with local updates.
• Demonstrate practical effectiveness across diverse datasets and models.
• Explore extensions to federated transfer learning and security considerations.

Proposed method

• Introduce Federated Stochastic Block Coordinate Descent (FedBCD) where parties share only a single value per sample during communication.
• Allow multiple local updates (Q) between communications to reduce rounds, forming parallel FedBCD-p and sequential FedBCD-s variants.
• Derive gradient updates using local information H and partial gradients: g_k(Θ;S) = ∇_k f(H_{-k}, θ_k; S) + λ∇γ(θ_k).
• Provide convergence analysis showing that with appropriate batch size, local iterations, and learning rate, the method attains O(1/√T) accuracy with O(√T) communication rounds.
• Present a proximal-augmented variant FedPBCD-p to improve stability for large Q.
• Discuss security considerations showing data privacy is preserved under the proposed leakage model.

Experimental results

Research questions

• RQ1 Can multiple local updates before communication reduce overall communication rounds in vertically partitioned collaborative learning?
• RQ2 What are the convergence guarantees of FedBCD when using stale information from partial gradients?
• RQ3 How do parallel and sequential local-update variants compare in practice on real datasets?
• RQ4 Does a proximal term improve convergence when local updates are large?
• RQ5 How does the framework perform under federated transfer learning and privacy-preserving settings?

Key findings

• FedBCD achieves an O(1/√T) convergence rate with O(√T) communication rounds under appropriate settings.
• In experiments on MIMIC-III and MNIST-CNN, increasing local updates Q reduces communication rounds with similar convergence to baseline SGD.
• FedBCD-p (parallel) and FedBCD-s (sequential) show comparable convergence, with FedBCD-s having higher wall-clock time due to sequential updates.
• Proximal FedPBCD-p stabilizes convergence for larger Q and can reach higher accuracy than FedBCD-p in large-Q regimes.
• Applying FedBCD with federated transfer learning tasks (NUS-FTL) demonstrates effective performance with reduced communication.
• In HE-enabled settings, larger Q reduces communication rounds but increases computation, with substantial overall time savings.