[Paper Review] Federated Progressive Sparsification (Purge, Merge, Tune)+

Dimitris Stripelis, Umang Gupta+2 more|arXiv (Cornell University)|Apr 26, 2022

Privacy-Preserving Technologies in Data4 citations

TL;DR

FedSparsify is a simple, iterative federated pruning method that progressively removes low-magnitude weights from neural networks during training, achieving 15-fold model size reduction, 4-fold inference speedup, and 3-fold communication cost reduction while maintaining or improving accuracy across diverse federated learning settings.

ABSTRACT

To improve federated training of neural networks, we develop FedSparsify, a sparsification strategy based on progressive weight magnitude pruning. Our method has several benefits. First, since the size of the network becomes increasingly smaller, computation and communication costs during training are reduced. Second, the models are incrementally constrained to a smaller set of parameters, which facilitates alignment/merging of the local models and improved learning performance at high sparsification rates. Third, the final sparsified model is significantly smaller, which improves inference efficiency and optimizes operations latency during encrypted communication. We show experimentally that FedSparsify learns a subnetwork of both high sparsity and learning performance. Our sparse models can reach a tenth of the size of the original model with the same or better accuracy compared to existing pruning and nonpruning baselines.

Motivation & Objective

• To develop a practical, iterative pruning strategy for federated learning that reduces model size and communication costs without sacrificing accuracy.
• To address the challenge of training extremely sparse neural networks in decentralized, non-IID data environments with limited computational and memory resources.
• To provide a strong, simple baseline for future research in federated model sparsification.
• To evaluate the impact of progressive pruning on convergence and performance in federated settings.
• To demonstrate that iterative magnitude-based pruning outperforms one-shot and dynamic pruning methods in extreme sparsity regimes.

Proposed method

• FedSparsify applies iterative, magnitude-based unstructured pruning at the end of each federated training round, removing the lowest-magnitude weights.
• The method alternates between pruning, merging (aggregating client models on the server), and fine-tuning (retraining on local data) in a progressive manner.
• Pruning is applied globally across clients using a global model aggregation strategy, ensuring consistent sparsity across the network.
• The process is repeated over multiple federation rounds, gradually increasing sparsity while maintaining model performance through fine-tuning.
• Sparsity is controlled via a target pruning ratio, with models pruned to 10% or less of the original size.
• The approach uses standard federated averaging (FedAvg) for model aggregation and fine-tuning to recover performance after pruning.

Experimental results

Research questions

• RQ1 Can iterative, magnitude-based pruning in federated learning achieve high sparsity (e.g., 99%) while preserving model accuracy?
• RQ2 How does progressive pruning compare to one-shot or dynamic pruning in terms of communication efficiency and inference speed?
• RQ3 Does the regularization effect of gradual pruning improve generalization in non-IID federated settings?
• RQ4 Can FedSparsify outperform non-pruning baselines (e.g., FedAvg) in terms of model efficiency and communication cost?
• RQ5 What is the impact of global vs. local pruning strategies on performance at extreme sparsity levels?

Key findings

• FedSparsify achieves an average 15-fold reduction in model size, with sparse models at 99% sparsity maintaining 7–9% accuracy loss compared to dense models.
• The method delivers a 4-fold improvement in inference throughput, with 4x and 8x speedup for CNN and VGG models at 99% sparsity, respectively.
• Communication costs are reduced by a factor of 3 compared to non-pruning baselines, due to fewer parameters exchanged per round.
• At 100 clients with non-IID data, FedSparsify-Global outperforms FedSparsify-Local and even dense baselines like FedAvg and FedProx.
• One-shot pruning with fine-tuning fails to generalize at 99% sparsity, while FedSparsify maintains performance through gradual pruning.
• The regularization effect of progressive pruning enhances model generalization, especially in data-scarce, non-IID environments.