[Paper Review] Throughput-Optimal Topology Design for Cross-Silo Federated Learning

Othmane Marfoq, Chuan Xu+2 more|arXiv (Cornell University)|Oct 23, 2020

Privacy-Preserving Technologies in Data99 references50 citations

TL;DR

This paper designs topologies for cross-silo federated learning to maximize system throughput using max-plus linear systems, showing significant training speedups over server-centric and MATCHA-based approaches.

ABSTRACT

Federated learning usually employs a client-server architecture where an orchestrator iteratively aggregates model updates from remote clients and pushes them back a refined model. This approach may be inefficient in cross-silo settings, as close-by data silos with high-speed access links may exchange information faster than with the orchestrator, and the orchestrator may become a communication bottleneck. In this paper we define the problem of topology design for cross-silo federated learning using the theory of max-plus linear systems to compute the system throughput---number of communication rounds per time unit. We also propose practical algorithms that, under the knowledge of measurable network characteristics, find a topology with the largest throughput or with provable throughput guarantees. In realistic Internet networks with 10 Gbps access links for silos, our algorithms speed up training by a factor 9 and 1.5 in comparison to the master-slave architecture and to state-of-the-art MATCHA, respectively. Speedups are even larger with slower access links.

Motivation & Objective

• Motivation: improve federated learning efficiency in cross-silo settings by leveraging fast inter-silo links.
• Goal: design communication topologies that maximize training throughput (rounds per time unit) while respecting overlay connectivity.
• Approach: integrate network measurements into topology design to minimize cycle time of the max-plus system.
• Outcome: provide algorithms with optimal or near-optimal guarantees and validate speedups on realistic network topologies.

Proposed method

• Model the training process as a synchronous DPASGD with local updates and neighbor communications.
• Define delay per overlay edge as d_o(i,j)=s·T_c(i)+l(i,j)+M/A(i′,j′) using underlay, connectivity, and overlay graphs.
• Formulate cycle time via max-plus algebra: τ(G_o)=max_γ d_o(γ)/|γ| and throughput as 1/τ(G_o).
• Propose topology design algorithms under edge-capacitated and node-capacitated settings (MCT problem).
• Provide approximation and optimality results: MST via Prim’s for edge-capacitated undirected overlays; 3N-approx using Christofides for Euclidean edge-capacitated graphs; 6-approx for certain node-capacitated Euclidean cases; NP-hardness results for directed overlays.
• Show practical performance comparisons against STAR and MATCHA/MATCHA+ overlays.

Experimental results

Research questions

• RQ1 How to design an overlay G_o within a connectivity graph G_c to minimize cycle time and maximize throughput in cross-silo FL?
• RQ2 What are the algorithmic guarantees (optimality/approximation) for Mct under edge-capacitated vs node-capacitated, and undirected vs directed overlays?
• RQ3 How does the proposed topology design impact training time and convergence when accounting for underlay delays, computation times, and queueing?
• RQ4 Do throughput-oriented topologies yield faster wall-clock training than server-centered or spectral-optimized overlays in realistic networks?

Key findings

• Overlays designed to maximize throughput achieve faster training times than STAR and often than MATCHA/MATCHA+ across multiple networks.
• RING, MST, and δ-MBST topologies provide substantial cycle-time reductions, with RING up to 2N times faster than STAR in slow-access regimes.
• In iNaturalist experiments, overlays designed from underlay/connectivity data yield significant cycle-time improvements, translating to large wall-clock speedups.
• For slow access links, low-degree overlays (e.g., RING, MST, δ-MBST) outperform high-degree overlays due to reduced per-round delays.
• MATCHA+ can surpass some baselines but requires underlay knowledge; throughput-focused designs without underlay assumptions still outperform in practice.