[Paper Review] CosSGD: Nonlinear Quantization for Communication-efficient Federated Learning.

Yang He, Maximilian Zenk+1 more|arXiv (Cornell University)|Dec 15, 2020

Privacy-Preserving Technologies in Data46 references4 citations

TL;DR

This paper proposes CosSGD, a nonlinear quantization method for communication-efficient federated learning that compresses stochastic gradients using cosine-based quantization. By reducing gradient communication by up to three orders of magnitude, CosSGD maintains model convergence and accuracy across image classification and brain tumor segmentation tasks on MNIST, CIFAR-10, and BraTS datasets.

ABSTRACT

Federated learning facilitates learning across clients without transferring local data on these clients to a central server. Despite the success of the federated learning method, it remains to improve further w.r.t communicating the most critical information to update a model under limited communication conditions, which can benefit this learning scheme into a wide range of application scenarios. In this work, we propose a nonlinear quantization for compressed stochastic gradient descent, which can be easily utilized in federated learning. Based on the proposed quantization, our system significantly reduces the communication cost by up to three orders of magnitude, while maintaining convergence and accuracy of the training process to a large extent. Extensive experiments are conducted on image classification and brain tumor semantic segmentation using the MNIST, CIFAR-10 and BraTS datasets where we show state-of-the-art effectiveness and impressive communication efficiency.

Motivation & Objective

• Address the challenge of high communication overhead in federated learning, especially in bandwidth-constrained environments.
• Improve communication efficiency without sacrificing model convergence or accuracy.
• Develop a practical, deployable quantization scheme compatible with existing federated learning frameworks.
• Enable broader application of federated learning in real-world scenarios with limited network resources.

Proposed method

• Propose a nonlinear quantization function based on cosine transformation to map stochastic gradients into a compressed representation.
• Design a quantization strategy that adaptively scales gradients using cosine-based intervals to preserve gradient direction and magnitude.
• Integrate the quantization mechanism into the compressed stochastic gradient descent (SGD) framework for federated learning.
• Maintain model convergence by ensuring quantized gradients retain sufficient information for effective parameter updates.
• Apply the method end-to-end in federated training pipelines across multiple datasets and model architectures.
• Use a symmetric quantization scheme with dynamic range adjustment based on gradient statistics to enhance precision.

Experimental results

Research questions

• RQ1 Can nonlinear quantization significantly reduce communication costs in federated learning without degrading model performance?
• RQ2 How does CosSGD compare to linear quantization methods in terms of convergence speed and final accuracy?
• RQ3 To what extent can gradient compression be achieved while preserving model generalization in diverse tasks?
• RQ4 Does the proposed method maintain robustness across different data distributions and model architectures?

Key findings

• CosSGD reduces communication cost by up to three orders of magnitude (1000x) compared to full-precision training.
• The method maintains model accuracy within 1% of full-precision baseline on MNIST and CIFAR-10 image classification tasks.
• On the BraTS brain tumor segmentation dataset, CosSGD achieved comparable segmentation performance to full-precision training.
• The proposed quantization enables stable convergence across multiple federated learning rounds with minimal degradation.
• CosSGD outperforms existing linear quantization baselines in communication efficiency and model accuracy trade-offs.
• The method is effective across diverse model architectures and data distributions, demonstrating broad applicability.