[Paper Review] Compression with Exact Error Distribution for Federated Learning

Mahmoud Hegazy, Rémi Leluc+2 more|arXiv (Cornell University)|Oct 31, 2023

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper proposes novel compression schemes for federated learning that achieve exact error distributions—such as Gaussian or Laplace—through layered quantizers and an aggregate Gaussian mechanism. By ensuring precise noise distributions, the method enables compression-for-free in differential privacy and recovers or improves upon standard techniques like Langevin dynamics and randomized smoothing with optimal convergence rates.

ABSTRACT

Compression schemes have been extensively used in Federated Learning (FL) to reduce the communication cost of distributed learning. While most approaches rely on a bounded variance assumption of the noise produced by the compressor, this paper investigates the use of compression and aggregation schemes that produce a specific error distribution, e.g., Gaussian or Laplace, on the aggregated data. We present and analyze different aggregation schemes based on layered quantizers achieving exact error distribution. We provide different methods to leverage the proposed compression schemes to obtain compression-for-free in differential privacy applications. Our general compression methods can recover and improve standard FL schemes with Gaussian perturbations such as Langevin dynamics and randomized smoothing.

Motivation & Objective

• To address the limitation of standard compression schemes that assume bounded variance but ignore the shape of compression noise.
• To design aggregation mechanisms that produce a specific, exact error distribution (e.g., Gaussian) on the final model update.
• To enable 'compression-for-free' in differential privacy by leveraging the exact error distribution for privacy guarantees.
• To improve convergence rates in non-smooth federated optimization via exact error distribution-based randomized smoothing.
• To ensure compatibility with secure aggregation by designing homomorphic compression schemes for less-trusted servers.

Proposed method

• Uses layered quantizers (direct and shifted) to achieve exact error distributions in individual compression mechanisms.
• Introduces the aggregate Gaussian mechanism, a novel homomorphic compression scheme ensuring the final aggregated output has exactly Gaussian error distribution.
• Applies the proposed quantizers to replace stochastic sampling in randomized smoothing, enabling exact recovery of Distributed Randomized Smoothing (DRS) algorithms.
• Designs compression schemes that are compatible with secure aggregation (SecAgg), allowing clients to send compressed messages that can be homomorphically aggregated before decoding.
• Leverages the exact error distribution to satisfy differential privacy constraints without additional noise, achieving 'compression-for-free' in DP settings.
• Analyzes communication costs of the proposed schemes, showing efficiency in both individual and aggregate compression settings.

Experimental results

Research questions

• RQ1 Can we design compression schemes that produce a specific, exact continuous error distribution (e.g., Gaussian or Laplace) rather than relying on bounded variance assumptions?
• RQ2 What are the communication costs and practical benefits of achieving exact error distributions in federated averaging and aggregation?
• RQ3 How can exact error distribution compression enable 'compression-for-free' in differential privacy applications?
• RQ4 Can such schemes be used to recover or improve upon existing methods like Langevin dynamics and randomized smoothing in non-smooth optimization?
• RQ5 How can the compression mechanism be made homomorphic to support secure aggregation in less-trusted server settings?

Key findings

• The proposed aggregate Gaussian mechanism ensures the final aggregated model update has exactly Gaussian error distribution, enabling precise differential privacy guarantees.
• The method achieves 'compression-for-free' in differential privacy by reusing the compression noise as the privacy-protecting noise, eliminating the need for additional perturbations.
• The use of layered quantizers with exact error distribution allows for exact recovery of the Distributed Randomized Smoothing (DRS) algorithm, achieving optimal convergence rates for non-smooth objectives.
• The aggregate Gaussian mechanism is homomorphic, making it compatible with secure aggregation protocols like SecAgg, thus enabling privacy-preserving aggregation even when the server is less trusted.
• Communication costs of the proposed schemes are analyzed and shown to be efficient, with the aggregate mechanism offering a favorable trade-off between error distribution control and bandwidth usage.
• The framework generalizes and improves upon standard FL schemes with Gaussian perturbations, such as those used in Langevin dynamics and randomized smoothing, by providing exact error control and tighter theoretical analysis.