[Paper Review] FedNew: A Communication-Efficient and Privacy-Preserving Newton-Type Method for Federated Learning

Anis Elgabli, Chaouki Ben Issaid+4 more|arXiv (Cornell University)|Jun 17, 2022

Stochastic Gradient Optimization Techniques4 citations

TL;DR

FedNew proposes a communication-efficient and privacy-preserving federated learning framework that replaces direct Hessian and gradient transmission with an alternating ADMM-based approximation of the inverse Hessian-gradient product. By using only one ADMM step per iteration and stochastic quantization, FedNew achieves faster convergence than first-order methods while preserving client privacy and reducing communication costs by up to 10× compared to exact Hessian-based methods.

ABSTRACT

Newton-type methods are popular in federated learning due to their fast convergence. Still, they suffer from two main issues, namely: low communication efficiency and low privacy due to the requirement of sending Hessian information from clients to parameter server (PS). In this work, we introduced a novel framework called FedNew in which there is no need to transmit Hessian information from clients to PS, hence resolving the bottleneck to improve communication efficiency. In addition, FedNew hides the gradient information and results in a privacy-preserving approach compared to the existing state-of-the-art. The core novel idea in FedNew is to introduce a two level framework, and alternate between updating the inverse Hessian-gradient product using only one alternating direction method of multipliers (ADMM) step and then performing the global model update using Newton's method. Though only one ADMM pass is used to approximate the inverse Hessian-gradient product at each iteration, we develop a novel theoretical approach to show the converging behavior of FedNew for convex problems. Additionally, a significant reduction in communication overhead is achieved by utilizing stochastic quantization. Numerical results using real datasets show the superiority of FedNew compared to existing methods in terms of communication costs.

Motivation & Objective

• Address the high communication cost and privacy leakage inherent in second-order federated learning methods that require transmission of Hessian and gradient information.
• Develop a framework that enables fast convergence using second-order information without exposing raw gradients or Hessians to the parameter server.
• Achieve communication efficiency comparable to first-order methods while maintaining the convergence speed advantages of Newton-type methods.
• Ensure privacy by hiding both gradient and Hessian data through a two-level optimization framework based on ADMM.
• Integrate stochastic quantization to further reduce communication overhead without sacrificing convergence performance.

Proposed method

• Introduce a two-level optimization framework: an inner level approximates the inverse Hessian-gradient product using only one ADMM step per iteration, and an outer level performs a Newton-like update.
• Use ADMM to solve the inner-level problem efficiently, enabling fast convergence without full Hessian computation or transmission.
• Approximate the inverse Hessian-gradient product as $(\nabla^2 f(x^k))^{-1} \nabla f(x^k)$ or a fixed initial Hessian version $(\nabla^2 f(x^0))^{-1} \nabla f(x^k)$ to reduce computational cost.
• Apply stochastic quantization to compress model updates, significantly reducing the number of transmitted bits per client.
• Design the algorithm so that clients never send raw gradients or Hessians—only compressed, privacy-preserving approximations.
• Prove asymptotic convergence to inexact Newton directions under convexity assumptions, with empirical validation showing robust performance across real datasets.

Experimental results

Research questions

• RQ1 Can a second-order federated learning method achieve fast convergence without transmitting raw gradients or Hessians, thus preserving client privacy?
• RQ2 Can a single ADMM step per iteration provide sufficient accuracy to approximate the inverse Hessian-gradient product while maintaining convergence?
• RQ3 To what extent can communication costs be reduced through quantization while preserving convergence speed and model accuracy?
• RQ4 How does the performance of FedNew compare to first-order (FedGD) and exact Hessian-based (Newton Zero) methods in terms of communication rounds and bits?
• RQ5 Does using a fixed Hessian (e.g., from the initial iteration) maintain convergence speed while reducing computational overhead?

Key findings

• FedNew-(r=1), which updates the Hessian at every iteration, converges faster than FedGD and matches the convergence speed of Newton Zero, achieving optimal performance with minimal communication rounds.
• FedNew-(r=0), which uses a fixed Hessian from the initial iteration, achieves convergence speed comparable to Newton Zero while preserving privacy and reducing computation by up to 10×.
• FedNew-(r=0.1), which updates the Hessian every 10 iterations, achieves convergence speed nearly identical to FedNew-(r=1) while reducing computation by a factor of 10.
• Q-FedNew, the quantized variant of FedNew, reduces communication bits by up to 10× compared to FedNew-(r=1) while maintaining the same optimality gap, demonstrating significant communication savings.
• For the w8a dataset, Q-FedNew-(r=1) achieves an optimality gap of $10^{-3}$ with almost 10× fewer transmitted bits than FedNew-(r=1), proving the effectiveness of quantization.
• Newton Zero incurs high initial communication cost due to full Hessian transmission, making it impractical for large-scale FL, whereas FedNew avoids this bottleneck entirely.