[Paper Review] Detailed comparison of communication efficiency of split learning and federated learning

Abhishek Singh, Praneeth Vepakomma+2 more|arXiv (Cornell University)|Sep 18, 2019

Privacy-Preserving Technologies in Data107 citations

TL;DR

The paper analyzes when split learning or federated learning achieve better communication efficiency under varying numbers of clients, model sizes, and data sizes, identifying regimes where each method excels.

ABSTRACT

We compare communication efficiencies of two compelling distributed machine learning approaches of split learning and federated learning. We show useful settings under which each method outperforms the other in terms of communication efficiency. We consider various practical scenarios of distributed learning setup and juxtapose the two methods under various real-life scenarios. We consider settings of small and large number of clients as well as small models (1M - 6M parameters), large models (10M - 200M parameters) and very large models (1 Billion-100 Billion parameters). We show that increasing number of clients or increasing model size favors split learning setup over the federated while increasing the number of data samples while keeping the number of clients or model size low makes federated learning more communication efficient.

Motivation & Objective

• Motivate understanding of how communication costs scale in split learning and federated learning.
• Characterize how factors like number of clients, model size, and data volume affect communication efficiency.
• Provide guidance on which method is more communication-efficient in practical scenarios.

Proposed method

• Define metrics for communication efficiency: per-client and total data transfer during training.
• Derive expressions for communication cost in split learning with and without client weight sharing.
• Derive communication cost for federated learning.
• Compare costs via a hyperbola-based boundary in model size vs. clients space.
• Use case illustrations to show practical regimes where each method wins.

Experimental results

Research questions

• RQ1 Under what conditions (in terms of number of clients K, model parameter size N, and data distribution p) does split learning outperform federated learning in communication efficiency?
• RQ2 How do variants of split learning (with vs. without client weight sharing) affect communication costs?
• RQ3 What practical scenarios (e.g., edge devices, healthcare, large institutions) illustrate the relative efficiency of the two methods?

Key findings

• Split learning becomes more communication-efficient as the number of clients increases and scales well with model size.
• Federated learning becomes more communication-efficient with larger data per client, especially when the number of clients or model size is small.
• With client weight sharing, communication overhead includes an ηN term, impacting efficiency depending on model size and smashed-layer size q.
• Without client weight sharing, split learning can outperform federated learning in high-client or high-parameter regimes.
• The paper characterizes a rectangular hyperbola boundary N = function(p, q, K, η) separating the regimes where each method dominates.
• Use-case figures (smart watches, healthcare, biobanks) illustrate where each approach is advantageous.