[论文解读] A(DP)$^2$SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

Jie Xu, Wei Zhang+1 more|arXiv (Cornell University)|Aug 21, 2020

Privacy-Preserving Technologies in Data参考文献 38被引用 8

一句话总结

该论文提出A(DP)²SGD，一种用于无中心服务器的分布式深度学习的差分隐私异步去中心化随机梯度下降框架，显著提升了隐私保护。通过将Rényi差分隐私应用于复合高斯机制，该方法实现了最优的O(1/√T)收敛速度，且模型精度与同步私有SGD相当，同时在异构环境中显著提升了训练速度。

ABSTRACT

As deep learning models are usually massive and complex, distributed learning is essential for increasing training efficiency. Moreover, in many real-world application scenarios like healthcare, distributed learning can also keep the data local and protect privacy. A popular distributed learning strategy is federated learning, where there is a central server storing the global model and a set of local computing nodes updating the model parameters with their corresponding data. The updated model parameters will be processed and transmitted to the central server, which leads to heavy communication costs. Recently, asynchronous decentralized distributed learning has been proposed and demonstrated to be a more efficient and practical strategy where there is no central server, so that each computing node only communicates with its neighbors. Although no raw data will be transmitted across different local nodes, there is still a risk of information leak during the communication process for malicious participants to make attacks. In this paper, we present a differentially private version of asynchronous decentralized parallel SGD (ADPSGD) framework, or A(DP)$^2$SGD for short, which maintains communication efficiency of ADPSGD and prevents the inference from malicious participants. Specifically, R{é}nyi differential privacy is used to provide tighter privacy analysis for our composite Gaussian mechanisms while the convergence rate is consistent with the non-private version. Theoretical analysis shows A(DP)$^2$SGD also converges at the optimal $\mathcal{O}(1/\sqrt{T})$ rate as SGD. Empirically, A(DP)$^2$SGD achieves comparable model accuracy as the differentially private version of Synchronous SGD (SSGD) but runs much faster than SSGD in heterogeneous computing environments.

研究动机与目标

• 解决在无中心服务器的异步去中心化训练中可能存在的隐私泄露风险。
• 在去中心化、异构计算环境中保持通信效率与可扩展性。
• 通过差分隐私提供强隐私保障，同时不牺牲收敛速度或模型精度。
• 实现在医疗等实际应用中私有分布式学习的实用化部署。

提出的方法

• 提出A(DP)²SGD，一种异步去中心化SGD（ADPSGD）的差分隐私变体，用于实现具有隐私保护的去中心化训练。
• 将Rényi差分隐私（RDP）应用于复合高斯机制，以实现比标准(ε,δ)-DP更紧致的隐私分析。
• 利用基于G-Lipschitz梯度和批量大小的敏感性边界，向梯度中注入校准后的高斯噪声。
• 通过T轮迭代中RDP机制的序列组合，推导出端到端的隐私预算(ε,δ)。
• 采用无放回采样以降低噪声方差，改善隐私-效用权衡。
• 通过RDP到(ε,δ)-DP的转换推导隐私参数，确保在异构数据分布下的正式隐私保障。

实验结果

研究问题

• RQ1 差分隐私能否在不降低收敛性或通信效率的前提下，有效集成到异步去中心化SGD中？
• RQ2 在去中心化、异构环境中，Rényi差分隐私如何优于标准(ε,δ)-DP实现更优的隐私分析？
• RQ3 在异步更新下，所提出的差分隐私去中心化SGD的收敛速率如何？
• RQ4 在实践中，A(DP)²SGD的模型精度与差分隐私同步SGD（SSGD）相比如何？
• RQ5 在异构计算环境中，A(DP)²SGD能否在保持相似隐私和精度的前提下，实现比SSGD更快的训练速度？

主要发现

• A(DP)²SGD实现了最优的O(1/√T)收敛速率，即使在引入差分隐私后，仍与非私有SGD保持一致。
• 该框架提供了(ε,δ)-差分隐私，其隐私预算ε由RDP组合与转换推导得出，确保了正式的隐私保障。
• 实验结果表明，A(DP)²SGD在CIFAR-10和SWB300数据集上实现了与差分隐私同步SGD相当的模型精度。
• 由于避免了同步瓶颈，A(DP)²SGD在异构环境中显著快于SSGD。
• Rényi差分隐私的使用实现了更紧致的隐私分析，允许更低的噪声方差和更优的效用-隐私权衡。
• 在异构数据分布和不同工作节点参与度下，该框架依然保持鲁棒性，表现为在多种设置下性能稳定。