[Paper Review] Improving the Privacy and Accuracy of ADMM-Based Distributed Algorithms

Xueru Zhang, Mohammad Mahdi Khalili+1 more|arXiv (Cornell University)|Jun 6, 2018

Privacy-Preserving Technologies in Data25 references41 citations

TL;DR

The paper introduces a penalty-perturbation method (private M-ADMM with PP) to achieve differential privacy in ADMM-based distributed learning, proving convergence and showing improved accuracy and privacy over prior dual/primal perturbation approaches through experiments on the Adult dataset.

ABSTRACT

Alternating direction method of multiplier (ADMM) is a popular method used to design distributed versions of a machine learning algorithm, whereby local computations are performed on local data with the output exchanged among neighbors in an iterative fashion. During this iterative process the leakage of data privacy arises. A differentially private ADMM was proposed in prior work (Zhang & Zhu, 2017) where only the privacy loss of a single node during one iteration was bounded, a method that makes it difficult to balance the tradeoff between the utility attained through distributed computation and privacy guarantees when considering the total privacy loss of all nodes over the entire iterative process. We propose a perturbation method for ADMM where the perturbed term is correlated with the penalty parameters; this is shown to improve the utility and privacy simultaneously. The method is based on a modified ADMM where each node independently determines its own penalty parameter in every iteration and decouples it from the dual updating step size. The condition for convergence of the modified ADMM and the lower bound on the convergence rate are also derived.

Motivation & Objective

• Motivate privacy concerns in ADMM-based distributed learning where data are dispersed across nodes.
• Propose a modified ADMM (M-ADMM) allowing each node to privately choose its penalty parameter per iteration.
• Introduce a penalty perturbation (PP) mechanism to achieve ε-differential privacy over the iterative process.
• Prove convergence and establish a lower bound on the convergence rate for the private algorithm.
• Demonstrate via experiments that increasing private penalties can improve both privacy guarantees and utility compared to prior approaches.

Proposed method

• Formulate a modified ADMM (M-ADMM) where each node independently selects its penalty parameter ηi(t+1) each iteration.
• Derive the simplified ADMM updates under node-private penalties: fi(t+1) via a perturbed objective (Eq. 10) and λi(t+1) (Eq. 11).
• Prove convergence of M-ADMM to the ERM optimum under standard convexity assumptions and network Laplacian properties (Theorems 3.1, 3.2).
• Introduce a penalty-perturbation (PP) scheme by adding a random vector εi(t+1) to the penalty term, leading to a private primal update (Eq. 25).
• Define a β-differential privacy guarantee for the iterative distributed process (Definition 4.1 and Theorem 4.1).
• Provide Algorithm 1 for PP, including parameter choice conditions to bound total privacy loss (β) and ensure convergence.

Experimental results

Research questions

• RQ1 Can a node-private penalty parameter in each iteration be used to preserve privacy in ADMM without sacrificing convergence?
• RQ2 How does a penalty-perturbation mechanism impact the trade-off between privacy loss and algorithm utility over the entire iterative process?
• RQ3 Does increasing private penalties over iterations enhance both privacy guarantees and accuracy compared to existing DP-ADMM methods?
• RQ4 What are the convergence conditions and rate bounds for the proposed private M-ADMM under standard convexity and Lipschitz assumptions?

Key findings

• The modified ADMM with node-private penalties converges to the ERM optimum under convexity and boundedness assumptions (Theorems 3.1 and 3.2).
• Penalty perturbation coupled with private ηi(t) improves both privacy and accuracy, with experiments showing better stability and higher accuracy than prior dual-variable perturbation (DVP) methods.
• Increasing the private penalty parameters over iterations can enhance robustness to noise, leading to improved accuracy and privacy together (Figures 1–2).
• The PP method achieves β-differential privacy over T iterations with a bound that scales with node data size and iteration count (Theorem 4.1).
• Empirical results on the Adult dataset with logistic loss demonstrate that suitable PP settings outperform DVP in both accuracy and privacy in networks of varying sizes (Figures 2–4).
• The technique generalizes beyond classification to other ADMM-based distributed algorithms where convergence and privacy analyses hold.