[Paper Review] Reconciling Security and Communication Efficiency in Federated Learning

Karthik R. Prasad, Sayan Ghosh+4 more|arXiv (Cornell University)|Jul 26, 2022

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper proposes secure, communication-efficient compression techniques for federated learning by adapting scalar quantization, pruning, and product quantization to work with Secure Aggregation (SecAgg), enabling up to 40× uplink compression with minimal accuracy loss. It introduces Secure Indexing (SecInd), a SecAgg variant that supports extreme compression via product quantization while preserving privacy and security guarantees.

ABSTRACT

Cross-device Federated Learning is an increasingly popular machine learning setting to train a model by leveraging a large population of client devices with high privacy and security guarantees. However, communication efficiency remains a major bottleneck when scaling federated learning to production environments, particularly due to bandwidth constraints during uplink communication. In this paper, we formalize and address the problem of compressing client-to-server model updates under the Secure Aggregation primitive, a core component of Federated Learning pipelines that allows the server to aggregate the client updates without accessing them individually. In particular, we adapt standard scalar quantization and pruning methods to Secure Aggregation and propose Secure Indexing, a variant of Secure Aggregation that supports quantization for extreme compression. We establish state-of-the-art results on LEAF benchmarks in a secure Federated Learning setup with up to 40$ imes$ compression in uplink communication with no meaningful loss in utility compared to uncompressed baselines.

Motivation & Objective

• To address the fundamental incompatibility between standard uplink compression techniques (e.g., quantization, pruning) and Secure Aggregation (SecAgg), which is critical for privacy in cross-device federated learning.
• To enable high communication efficiency in federated learning without compromising security, particularly for uplink updates where bandwidth is constrained.
• To develop compression methods that maintain linearity in the decompression operator to preserve compatibility with SecAgg protocols.
• To introduce Secure Indexing (SecInd), a novel SecAgg variant that supports product quantization for extreme compression while maintaining strong security guarantees.
• To demonstrate state-of-the-art performance on LEAF benchmarks with minimal utility degradation under secure, compressed communication.

Proposed method

• Adapt scalar quantization and (random) pruning to SecAgg by imposing a linearity constraint on the decompression operator, ensuring compatibility without modifying the SecAgg protocol.
• Propose Secure Indexing (SecInd), a variant of SecAgg that supports product quantization by operating in the compressed domain and enabling secure aggregation of quantized indices.
• Use a TEE-based SecAgg framework where clients compress their model updates using a shared compression operator, encrypt the compressed updates with random masks, and send them to the server.
• Enable periodic updates of compression parameters (e.g., quantization levels, codebooks, pruning masks) broadcast by the server to reduce staleness and improve accuracy.
• Ensure that decompression is linear so that the server can recover the aggregate in the compressed domain and convert it back to the original domain with minimal error.
• Leverage existing SecAgg infrastructure (e.g., TEE-based) to maintain security while integrating compression, avoiding changes to the core protocol.

Experimental results

Research questions

• RQ1 Can standard compression techniques like scalar quantization and pruning be made compatible with Secure Aggregation in federated learning without breaking privacy guarantees?
• RQ2 What is the maximum achievable uplink compression rate under end-to-end security and minimal accuracy degradation in a SecAgg-compliant setup?
• RQ3 How can product quantization be integrated into SecAgg to enable extreme compression (e.g., 40×) while preserving security and utility?
• RQ4 What is the impact of periodic updates to compression parameters (e.g., codebooks, quantization levels) on model accuracy and staleness?
• RQ5 How does the choice of SecAgg bit-width affect overflow rates in scalar quantization, and what trade-offs exist between security and communication efficiency?

Key findings

• The proposed method achieves up to 40× compression in uplink communication on the FEMNIST benchmark with no meaningful loss in model utility compared to uncompressed baselines.
• Scalar quantization degrades significantly at high compression rates due to aggregation overflows, especially when the SecAgg bit-width is too small relative to the quantization bit-width.
• Pruning offers intermediate trade-offs between compression efficiency and accuracy, with performance improving when masks are refreshed periodically.
• Product quantization enables extreme compression with minimal accuracy loss, outperforming scalar quantization at high compression rates.
• The frequency of compression operator updates significantly affects accuracy, particularly for aggressive compression setups like large block sizes or small codebook sizes in product quantization.
• Overflows in scalar quantization can be minimized by setting the SecAgg bit-width to be at least ⌈log₂n_c⌉ greater than the quantization bit-width, where n_c is the number of clients.