[Paper Review] Learning Privately over Distributed Features: An ADMM Sharing Approach

Yaochen Hu, Peng Liu+2 more|arXiv (Cornell University)|Jul 17, 2019

Privacy-Preserving Technologies in Data44 references32 citations

TL;DR

Proposes an ADMM sharing framework for empirical risk minimization over vertically partitioned (distributed) features, enabling privacy by transmitting only a single value per sample and providing a differential privacy variant.

ABSTRACT

Distributed machine learning has been widely studied in order to handle exploding amount of data. In this paper, we study an important yet less visited distributed learning problem where features are inherently distributed or vertically partitioned among multiple parties, and sharing of raw data or model parameters among parties is prohibited due to privacy concerns. We propose an ADMM sharing framework to approach risk minimization over distributed features, where each party only needs to share a single value for each sample in the training process, thus minimizing the data leakage risk. We establish convergence and iteration complexity results for the proposed parallel ADMM algorithm under non-convex loss. We further introduce a novel differentially private ADMM sharing algorithm and bound the privacy guarantee with carefully designed noise perturbation. The experiments based on a prototype system shows that the proposed ADMM algorithms converge efficiently in a robust fashion, demonstrating advantage over gradient based methods especially for data set with high dimensional feature spaces.

Motivation & Objective

• Motivate learning when features are inherently distributed across parties and raw data cannot be shared due to privacy.
• Develop an ADMM-based algorithm that minimizes empirical risk with vertically partitioned data while sharing minimal information.
• Establish convergence guarantees for parallel ADMM sharing under non-convex losses and provide a differential privacy version.
• Show empirical evidence that the approach outperforms gradient-based methods on high-dimensional feature datasets.

Proposed method

• Reformulate empirical risk minimization with distributed features into a constrained problem using an auxiliary variable z.
• Use a parallel ADMM scheme where each party updates its local variable x_m and the central node updates z and the dual variable y.
• Each party shares only a single value per iteration, reducing data leakage risk.
• Prove convergence and iteration complexity for parallel ADMM under non-convex loss, extending prior Gauss-Seidel results.
• Introduce a differentially private ADMM variant by adding Gaussian noise to the shared messages to achieve (ε,δ)-DP.
• Provide a Lyapunov-based analysis for convergence and derive sensitivity-based noise scales to guarantee DP.

Experimental results

Research questions

• RQ1 Can an ADMM-based approach solve ERM with vertically partitioned data without sharing raw features or local models?
• RQ2 What convergence and iteration complexity guarantees can be established for parallel ADMM in the non-convex loss setting?
• RQ3 How can differential privacy be integrated into ADMM sharing to protect intermediate communications without crippling performance?
• RQ4 How does the ADMM sharing method compare to gradient-based methods on high-dimensional feature spaces?

Key findings

• The parallel ADMM sharing algorithm converges to stationary solutions under mild assumptions for non-convex losses.
• The method achieves convergence with iteration complexity characterized via a Lyapunov function, requiring a finite number of iterations to reach an ϵ-accurate solution.
• A differentially private version of ADMM sharing guarantees ε,δ-differential privacy per iteration and across T epochs using Gaussian noise calibrated to a sensitivity bound.
• Experiments indicate the ADMM sharing algorithm converges efficiently and is robust, particularly on datasets with large feature spaces, and can outperform gradient-based methods in high-dimensional settings.
• Differential privacy can be maintained with moderate perturbation without derailing convergence, improving privacy without sacrificing too much predictive accuracy.