[Paper Review] FedHybrid: A Hybrid Primal-Dual Algorithm Framework for Federated Optimization

Xiaochun Niu, Ermin Wei|arXiv (Cornell University)|Jun 2, 2021

Distributed Control Multi-Agent Systems22 references4 citations

TL;DR

FedHybrid proposes a novel hybrid primal-dual algorithm framework for federated optimization that enables heterogeneous clients to independently choose between gradient-type or Newton-type updates based on their computational capabilities. It achieves linear convergence for strongly convex functions and outperforms baseline methods like FedAvg, especially when some clients use second-order information.

ABSTRACT

We consider a multi-agent consensus optimization problem over a server-client (federated) network, where all clients are connected to a central server. Current distributed algorithms fail to capture the heterogeneity in clients' local computation capacities. Motivated by the generalized Method of Multipliers in centralized optimization, we derive an approximate Newton-type primal-dual method with a practical distributed implementation by utilizing the server-client topology. Then we propose a new primal-dual algorithm framework FedHybrid that allows different clients to perform various types of updates. Specifically, each client can choose to perform either gradient-type or Newton-type updates. We propose a novel analysis framework for primal-dual methods and obtain a linear convergence rate of FedHybrid for strongly convex functions, regardless of clients' choices of gradient-type or Newton-type updates. Numerical studies are provided to demonstrate the efficacy of our method in practice. To the best of our knowledge, this is the first hybrid algorithmic framework allowing heterogeneous local updates for distributed consensus optimization with a provable convergence and rate guarantee.

Motivation & Objective

• To address the lack of flexible, provably convergent algorithms in federated learning that accommodate clients with varying computational capacities.
• To develop a distributed primal-dual method that integrates both first-order and second-order updates in a unified framework.
• To provide a theoretical convergence guarantee—specifically linear (Q-linear) convergence—for the proposed hybrid framework under strong convexity.
• To demonstrate practical efficacy through numerical experiments on synthetic and real-world datasets with non-IID data partitioning.

Proposed method

• The framework is built on a generalized Method of Multipliers, using a distributed approximation of the primal and dual Hessian to enable Newton-type updates.
• Clients are allowed to choose between gradient-type or Newton-type local updates based on their computational resources.
• The server coordinates consensus by enforcing $ x_0 = x_i $ across clients using a consensus constraint matrix $ W $.
• A novel analysis framework is developed to establish linear convergence for the last iterate, regardless of client update type choices.
• The algorithm uses a penalty parameter $ \rho $ to balance dual and primal updates, ensuring stability and convergence.
• The method is implemented in a server-client topology, where the server aggregates and coordinates updates without requiring raw data exchange.

Experimental results

Research questions

• RQ1 Can a federated optimization framework support heterogeneous clients using mixed first- and second-order update types while maintaining convergence?
• RQ2 What is the convergence rate of a hybrid primal-dual algorithm when clients independently choose between gradient and Newton updates?
• RQ3 How does the inclusion of Newton-type updates affect convergence speed compared to purely first-order methods in non-IID settings?
• RQ4 Can a provably convergent algorithm be designed for federated consensus optimization that leverages second-order information in a distributed manner?

Key findings

• FedHybrid achieves linear (Q-linear) convergence for strongly convex functions, regardless of whether clients use gradient-type or Newton-type updates.
• The method outperforms FedAvg in convergence speed, especially when a subset of clients performs Newton-type updates.
• When all clients use Newton-type updates (FedH-N variant), convergence performance is comparable to DiSCO, a state-of-the-art second-order method.
• Increasing the number of clients using Newton-type updates leads to faster overall convergence, demonstrating scalability of the hybrid approach.
• Numerical experiments on synthetic and real datasets (Boston housing, mushroom) confirm the method’s efficacy under non-IID data partitioning.