[Paper Review] Cross-Silo Federated Learning for Multi-Tier Networks with Vertical and Horizontal Data Partitioning

Anirban Das, Timothy Castiglia+2 more|arXiv (Cornell University)|Aug 19, 2021

Privacy-Preserving Technologies in Data35 references4 citations

TL;DR

This paper proposes Tiered Decentralized Coordinate Descent (TDCD), a communication-efficient federated learning algorithm for multi-tier networks with both vertical and horizontal data partitioning. Clients within each silo perform multiple local updates before sharing with their hub, and hubs exchange intermediate updates, achieving convergence with reduced communication overhead and theoretical guarantees on convergence rate dependence on local steps and vertical partitions.

ABSTRACT

We consider federated learning in tiered communication networks. Our network model consists of a set of silos, each holding a vertical partition of the data. Each silo contains a hub and a set of clients, with the silo's vertical data shard partitioned horizontally across its clients. We propose Tiered Decentralized Coordinate Descent (TDCD), a communication-efficient decentralized training algorithm for such two-tiered networks. The clients in each silo perform multiple local gradient steps before sharing updates with their hub to reduce communication overhead. Each hub adjusts its coordinates by averaging its workers' updates, and then hubs exchange intermediate updates with one another. We present a theoretical analysis of our algorithm and show the dependence of the convergence rate on the number of vertical partitions and the number of local updates. We further validate our approach empirically via simulation-based experiments using a variety of datasets and objectives.

Motivation & Objective

• To address the challenge of training global models in multi-tier networks where data is vertically partitioned across silos and horizontally partitioned within silos.
• To reduce communication overhead in federated learning by allowing multiple local updates before inter-silo communication.
• To develop a decentralized training algorithm that maintains convergence guarantees under complex data partitioning and communication constraints.
• To theoretically analyze the convergence rate of the proposed algorithm in terms of number of vertical partitions and local update steps.
• To empirically validate the approach using simulations across diverse datasets and objectives.

Proposed method

• TDCD operates in a two-tiered network: clients within each silo perform multiple local gradient steps before updating their hub.
• Each hub aggregates updates from its clients via coordinate averaging to maintain model consistency.
• Hubs exchange intermediate model updates with one another to synchronize progress across silos.
• The algorithm uses a decentralized coordinate descent framework, where each silo optimizes only its subset of model coordinates.
• Theoretical analysis bounds the convergence rate, showing dependence on the number of vertical partitions and local update steps.
• A key technical contribution is the derivation of convergence bounds under non-convex optimization with bounded gradients and noise.

Experimental results

Research questions

• RQ1 How can federated learning be efficiently scaled in multi-tier networks with both vertical and horizontal data partitioning?
• RQ2 What is the impact of multiple local updates on convergence and communication efficiency in vertical federated learning?
• RQ3 How does the number of vertical partitions (i.e., silos) affect the convergence rate of the global model?
• RQ4 Can a decentralized, communication-efficient algorithm maintain convergence guarantees in such complex network and data structures?
• RQ5 What is the trade-off between local computation and communication frequency in this setting?

Key findings

• TDCD achieves convergence with a theoretical bound showing that the convergence rate depends on the number of vertical partitions and the number of local updates per communication round.
• The algorithm reduces communication overhead by allowing multiple local updates before inter-silo communication, improving efficiency in high-latency networks.
• Theoretical analysis confirms that the method converges under non-convex objectives with bounded gradients and stochastic noise.
• The convergence rate is bounded by a term inversely proportional to the learning rate and local update count, with a diminishing return beyond a certain number of local steps.
• Empirical simulations validate the approach across multiple datasets, showing improved training efficiency and convergence stability.
• The method maintains model accuracy while minimizing data transfer, making it suitable for privacy-sensitive, multi-organizational settings like healthcare and finance.