[Paper Review] Collecting and Analyzing Multidimensional Data with Local Differential Privacy

Ning Wang, Xiaokui Xiao+6 more|arXiv (Cornell University)|Jun 28, 2019

Privacy-Preserving Technologies in Data42 references39 citations

TL;DR

The paper introduces two new LDP mechanisms (Piecewise Mechanism and Hybrid Mechanism) for collecting numeric and multidimensional data, analyzes their worst-case error, and applies them to LDP-compliant SGD for machine learning tasks.

ABSTRACT

Local differential privacy (LDP) is a recently proposed privacy standard for collecting and analyzing data, which has been used, e.g., in the Chrome browser, iOS and macOS. In LDP, each user perturbs her information locally, and only sends the randomized version to an aggregator who performs analyses, which protects both the users and the aggregator against private information leaks. Although LDP has attracted much research attention in recent years, the majority of existing work focuses on applying LDP to complex data and/or analysis tasks. In this paper, we point out that the fundamental problem of collecting multidimensional data under LDP has not been addressed sufficiently, and there remains much room for improvement even for basic tasks such as computing the mean value over a single numeric attribute under LDP. Motivated by this, we first propose novel LDP mechanisms for collecting a numeric attribute, whose accuracy is at least no worse (and usually better) than existing solutions in terms of worst-case noise variance. Then, we extend these mechanisms to multidimensional data that can contain both numeric and categorical attributes, where our mechanisms always outperform existing solutions regarding worst-case noise variance. As a case study, we apply our solutions to build an LDP-compliant stochastic gradient descent algorithm (SGD), which powers many important machine learning tasks. Experiments using real datasets confirm the effectiveness of our methods, and their advantages over existing solutions.

Motivation & Objective

• Motivate and address the gap in collecting multidimensional data under local differential privacy (LDP).
• Develop mechanisms that achieve better worst-case noise variance for numeric attributes under ε-LDP.
• Extend single-attribute mechanisms to multidimensional numeric and categorical data with strong privacy-utility guarantees.
• Provide a case study by building an LDP-compliant stochastic gradient descent (SGD) pipeline for machine learning tasks.

Proposed method

• Propose Piecewise Mechanism (PM) for a single numeric attribute under ε-LDP with bounded output and data-dependent piecewise noise.
• Develop Hybrid Mechanism (HM) that combines PM and Duchi et al.’s approach to minimize worst-case variance.
• Extend PM and HM to multidimensional data by selecting a subset of attributes and perturbing them with adjusted privacy budgets (Algorithm 4).
• Provide an extension to handle mixtures of numeric and categorical attributes (Section IV-C).
• Analyze worst-case variance of PM, HM, and existing methods, establishing asymptotic optimality for the multidimensional setting.
• Demonstrate an LDP-compliant SGD framework using the proposed mechanisms for linear/logistic regression and SVM.

Experimental results

Research questions

• RQ1 How can numeric data be collected under ε-local differential privacy with improved accuracy compared to Laplace or existing methods?
• RQ2 Can PM and HM be extended to multidimensional numeric and categorical data with asymptotically optimal error bounds?
• RQ3 How do the proposed mechanisms perform in practice for statistical estimation and gradient-based learning tasks under LDP?
• RQ4 What is the impact of attribute sampling (Algorithm 4) on privacy-utility tradeoffs in high-dimensional data?
• RQ5 Can the mechanisms be effectively integrated into an LDP-compliant SGD framework for common ML models?

Key findings

• PM achieves lower worst-case noise variance than Laplace, and often better than Duchi et al.’s method for ε ≥ ~1.29 on one-dimensional data.
• HM yields always no worse (and often better) worst-case variance than PM and Duchi et al.’s approach across ε ranges.
• Extensions to multidimensional numeric data (Algorithm 4) attain asymptotically optimal error (matching Duchi et al.’s bounds) with simpler implementation.
• The proposed multidimensional mechanisms apply to mixtures of numeric and categorical attributes with strong privacy guarantees.
• An LDP-compliant SGD algorithm built from PM/HM components demonstrates effectiveness on real datasets for tasks like linear/logistic regression and SVM.