[Paper Review] Scalable and Provably Accurate Algorithms for Differentially Private Distributed Decision Tree Learning

Kaiwen Wang, Travis Dick+1 more|arXiv (Cornell University)|Dec 19, 2020

Privacy-Preserving Technologies in Data24 references4 citations

TL;DR

This paper proposes DP-TopDown, a differentially private, top-down decision tree learning framework for distributed settings, using two implementations: NoisyCounts (global noise aggregation) and LocalRNM (local optimization to reduce communication and noise). It provides the first provably accurate utility guarantees for private decision trees, showing error decreases rapidly with larger datasets under the Weak Learning Assumption.

ABSTRACT

This paper introduces the first provably accurate algorithms for differentially private, top-down decision tree learning in the distributed setting (Balcan et al., 2012). We propose DP-TopDown, a general privacy preserving decision tree learning algorithm, and present two distributed implementations. Our first method NoisyCounts naturally extends the single machine algorithm by using the Laplace mechanism. Our second method LocalRNM significantly reduces communication and added noise by performing local optimization at each data holder. We provide the first utility guarantees for differentially private top-down decision tree learning in both the single machine and distributed settings. These guarantees show that the error of the privately-learned decision tree quickly goes to zero provided that the dataset is sufficiently large. Our extensive experiments on real datasets illustrate the trade-offs of privacy, accuracy and generalization when learning private decision trees in the distributed setting.

Motivation & Objective

• Address the challenge of training accurate decision trees in distributed environments while preserving strong differential privacy guarantees.
• Overcome the privacy-accuracy trade-off in distributed decision tree learning by designing efficient, private splitting mechanisms.
• Provide theoretical utility guarantees for differentially private top-down decision tree learning in both single-machine and distributed settings.
• Demonstrate empirically that private decision trees can generalize better than non-private ones in certain cases, especially with small or noisy datasets.

Proposed method

• Propose DP-TopDown, a family of differentially private decision tree algorithms based on the TopDown framework, using a private subroutine called PrivateSplit to approximate optimal splits.
• Implement NoisyCounts: each data holder publishes Laplace-noised class counts, which are aggregated to select splits, ensuring differential privacy at the cost of high noise and communication.
• Introduce LocalRNM: a heuristic that performs local optimization at each data holder before communication, significantly reducing the amount of noise and communication compared to NoisyCounts.
• Apply decay budgeting to allocate privacy budget dynamically between internal node splits and leaf labeling, improving accuracy by prioritizing critical decisions.
• Use entropy as the splitting criterion and apply the exponential mechanism or Laplace mechanism to ensure differential privacy during split selection.
• Leverage the Weak Learning Assumption and theoretical bounds to derive utility guarantees showing that generalization error converges to zero as dataset size increases.

Experimental results

Research questions

• RQ1 Can we design a differentially private, distributed decision tree learning algorithm that maintains high accuracy while ensuring strong privacy guarantees?
• RQ2 How does the allocation of privacy budget between internal splits and leaf labeling affect the final model accuracy in distributed decision tree learning?
• RQ3 Does injecting noise into the greedy splitting process improve generalization, particularly in cases where non-private models overfit?
• RQ4 Can we theoretically bound the generalization error of differentially private decision trees under the Weak Learning Assumption in both single-machine and distributed settings?
• RQ5 How do communication cost and noise level trade off in distributed private decision tree learning, and can local optimization reduce this trade-off?

Key findings

• DP-TopDown provides the first provably accurate utility guarantees for differentially private top-down decision tree learning in both single-machine and distributed settings, with error converging to zero as dataset size increases.
• LocalRNM outperforms NoisyCounts in terms of test accuracy across multiple real-world datasets (Adult, Bank, Creditcard, Skin), due to reduced noise and communication through local optimization.
• On the CTR dataset, private algorithms achieved higher test accuracy than non-private baselines for certain privacy parameters, suggesting that noise injection can help escape poor local optima in greedy learning.
• Larger training datasets consistently yield higher test accuracy and lower variance in private models, confirming theoretical expectations and the benefit of data diversity in private learning.
• The decay budgeting strategy improves performance by allocating more privacy budget to critical decisions, and fixing LeafPrivacyFraction at 0.5 yields stable results across experiments.
• Training accuracy increases with dataset size, even though smaller datasets are more prone to overfitting, indicating that private learning benefits from data scale and noise regularization.