[Paper Review] Castell: Scalable Joint Probability Estimation of Multi-dimensional Data Randomized with Local Differential Privacy

Hiroaki Kikuchi|arXiv (Cornell University)|Dec 3, 2022

Survey Sampling and Estimation Techniques4 citations

TL;DR

This paper proposes Castell, a scalable local differential privacy (LDP) scheme for estimating multi-dimensional joint probability distributions by independently randomizing each attribute and aggregating the resulting randomization matrices. The method enables efficient inversion of the aggregated matrix with linear computation cost and manageable storage, achieving significantly higher accuracy than prior state-of-the-art methods like LoPub, with average estimation errors of 0.0099 (truncated) and 0.0155 (hybrid), compared to 0.03 and 0.04 respectively.

ABSTRACT

Performing randomized response (RR) over multi-dimensional data is subject to the curse of dimensionality. As the number of attributes increases, the exponential growth in the number of attribute-value combinations greatly impacts the computational cost and the accuracy of the RR estimates. In this paper, we propose a new multi-dimensional RR scheme that randomizes all attributes independently, and then aggregates these randomization matrices into a single aggregated matrix. The multi-dimensional joint probability distributions are then estimated. The inverse matrix of the aggregated randomization matrix can be computed efficiently at a lightweight computation cost (i.e., linear with respect to dimensionality) and with manageable storage requirements. To overcome the limitation of accuracy, we propose two extensions to the baseline protocol, called {\em hybrid} and {\em truncated} schemes. Finally, we have conducted experiments using synthetic and major open-source datasets for various numbers of attributes, domain sizes, and numbers of respondents. The results using UCI Adult dataset give average distances between the estimated and the real (2 through 6-way) joint probability are $0.0099$ for {\em truncated} and $0.0155$ for {\em hybrid} schemes, whereas they are $0.03$ and $0.04$ for LoPub, which is the state-of-the-art multi-dimensional LDP scheme.

Motivation & Objective

• Address the curse of dimensionality in multi-dimensional local differential privacy (LDP) by overcoming exponential growth in domain size and computational cost.
• Preserve cross-attribute dependencies in high-dimensional data while maintaining scalability and low computational overhead.
• Overcome the accuracy limitations of independent randomization (RR-Independent) and the infeasibility of joint randomization (RR-Joint) in high dimensions.
• Develop a practical, efficient, and accurate method for estimating multi-dimensional joint probability distributions under LDP guarantees.
• Provide a framework that balances privacy budget, dimensionality, and estimation accuracy through adaptive hybrid and truncated schemes.

Proposed method

• Propose a new RR scheme, RR-Ind-Joint, which independently randomizes each attribute and aggregates the resulting randomization matrices into a single aggregated matrix.
• Introduce the Castell algorithm to efficiently invert the aggregated randomization matrix with computation cost linear in dimensionality and storage cost of d^w, matching cross-tabulation requirements.
• Design a hybrid RR scheme that dynamically switches between RR-Independent and RR-Ind-Joint based on privacy budget, dimensionality, and number of respondents.
• Develop a truncated scheme that limits the number of non-zero entries in the randomization matrix to reduce noise and improve accuracy.
• Derive theoretical upper bounds for estimation error in both RR-Ind-Joint and RR-Independent schemes to guide parameter selection.
• Use synthetic and real-world datasets (including UCI Adult) to evaluate performance across varying dimensions, domain sizes, and respondent counts.

Experimental results

Research questions

• RQ1 Can independent randomization of attributes in multi-dimensional data be combined with matrix aggregation to achieve accurate joint probability estimation under LDP?
• RQ2 Does the Castell algorithm enable efficient inversion of the aggregated randomization matrix with linear time complexity and manageable storage?
• RQ3 How does the hybrid scheme improve estimation accuracy by adaptively switching between independent and joint randomization strategies?
• RQ4 To what extent does the truncated scheme reduce estimation error compared to baseline RR-Ind-Joint and RR-Independent?
• RQ5 How does Castell compare in accuracy to state-of-the-art LDP schemes like LoPub and LoCop across different numbers of attributes and privacy budgets?

Key findings

• The Castell algorithm enables efficient inversion of the aggregated randomization matrix with computation cost linear in dimensionality and storage cost of d^w, making it scalable to high-dimensional data.
• On the UCI Adult dataset, the truncated scheme achieved an average estimation error of 0.0099 for 2–6-way joint probabilities, outperforming LoPub’s 0.03.
• The hybrid scheme achieved an average error of 0.0155, significantly lower than LoPub’s 0.04, demonstrating improved accuracy under the same conditions.
• The proposed hybrid scheme effectively balances scalability and accuracy by switching between RR-Independent and RR-Ind-Joint based on input parameters such as privacy budget and dimensionality.
• Theoretical upper bounds on estimation error were derived for both RR-Ind-Joint and RR-Independent, providing a foundation for adaptive parameter selection.
• Experimental results show that Castell maintains high accuracy across diverse datasets and configurations, including synthetic and open-source real-world data.