[Paper Review] Communication Efficient Federated Learning over Multiple Access Channels

Wei-Ting Chang, Ravi Tandon|arXiv (Cornell University)|Jan 23, 2020

Privacy-Preserving Technologies in Data27 references52 citations

TL;DR

The paper designs digital gradient transmission over a Gaussian MAC for federated learning, introducing MAC-aware stochastic gradient quantization that allocates per-user quantization budgets based on gradient informativeness and channel capacity to improve convergence. It analyzes convergence and provides an optimization framework, with a two-user example and experiments showing MAC-aware quantization outperforms uniform and other digital schemes.

ABSTRACT

In this work, we study the problem of federated learning (FL), where distributed users aim to jointly train a machine learning model with the help of a parameter server (PS). In each iteration of FL, users compute local gradients, followed by transmission of the quantized gradients for subsequent aggregation and model updates at PS. One of the challenges of FL is that of communication overhead due to FL's iterative nature and large model sizes. One recent direction to alleviate communication bottleneck in FL is to let users communicate simultaneously over a multiple access channel (MAC), possibly making better use of the communication resources. In this paper, we consider the problem of FL learning over a MAC. In particular, we focus on the design of digital gradient transmission schemes over a MAC, where gradients at each user are first quantized, and then transmitted over a MAC to be decoded individually at the PS. When designing digital FL schemes over MACs, there are new opportunities to assign different amount of resources (such as rate or bandwidth) to different users based on a) the informativeness of the gradients at each user, and b) the underlying channel conditions. We propose a stochastic gradient quantization scheme, where the quantization parameters are optimized based on the capacity region of the MAC. We show that such channel aware quantization for FL outperforms uniform quantization, particularly when users experience different channel conditions, and when have gradients with varying levels of informativeness.

Motivation & Objective

• Motivate reducing communication overhead in federated learning by exploiting multiple access channels (MACs).
• Develop a digital gradient transmission scheme with per-user quantization budgets tuned to MAC capacity and gradient informativeness.
• Formulate and solve an optimization problem to allocate quantization budgets under MAC capacity constraints.
• Provide theoretical convergence guarantees and practical insights via a two-user example and experiments on MNIST.

Proposed method

• Model FL with M users sending quantized gradients over a Gaussian MAC to a parameter server (PS).
• Introduce stochastic multi-level gradient quantization per user with dynamic range based budgets k_m determining quantization levels k_m.
• Prove unbiased gradient estimates with bounded variance and derive convergence bounds depending on gradient dynamic ranges and quantization budgets.
• Formulate MAC-aware optimization (problem P) to minimize sum of variances across users under MAC capacity constraints.
• Solve the relaxed convex version of the 2-user case analytically to gain insights into budget allocation as a function of gradient ranges and MAC capacities.
• Demonstrate via MNIST experiments that MAC-aware quantization outperforms uniform allocation and other digital schemes.

Experimental results

Research questions

• RQ1 How should transmission rates (quantization budgets) be allocated among users transmitting over a MAC to optimize federated learning convergence?
• RQ2 What is the impact of gradient informativeness (dynamic range) and MAC capacity on optimal per-user quantization budgets?
• RQ3 Can a convex relaxation of the per-iteration budget optimization yield practical, near-optimal allocations for multi-user FL over MACs?
• RQ4 Do MAC-aware schemes outperform uniform quantization and existing digital schemes in practice?

Key findings

• MAC-aware gradient quantization allocates more rate to users with more informative gradients (higher dynamic range).
• The convergence bound depends on the sum of per-user gradient quantization variances, which are controlled by dynamic ranges and quantization levels.
• For two users, the optimal budgets can be characterized by a closed-form-like condition linking gradient dynamic ranges and MAC capacities.
• Experiments on MNIST show MAC-aware quantization outperforms uniform allocation, a prior digital scheme, SignSGD, and TernGrad, approaching full-resolution performance.
• Increasing total power (larger MAC capacity) improves training accuracy, and the scheme adapts quantization budgets accordingly.
• The approach maintains unbiased gradient estimates with bounded variance under the MAC constraints.