[Paper Review] Differential Private Stack Generalization with an Application to Diabetes Prediction

Quanming Yao, Xiawei Guo+5 more|arXiv (Cornell University)|Nov 23, 2018

Privacy-Preserving Technologies in Data33 references4 citations

TL;DR

This paper proposes differentially private stacking (PST) to enhance privacy-preserving logistic regression by leveraging ensemble learning through sample- or feature-based data partitioning. It demonstrates theoretically and empirically that feature-based partitioning reduces sample complexity and improves prediction performance, especially when incorporating feature importance, achieving state-of-the-art results in cross-organizational diabetes prediction while preserving privacy.

ABSTRACT

To meet the standard of differential privacy, noise is usually added into the original data, which inevitably deteriorates the predicting performance of subsequent learning algorithms. In this paper, motivated by the success of improving predicting performance by ensemble learning, we propose to enhance privacy-preserving logistic regression by stacking. We show that this can be done either by sample-based or feature-based partitioning. However, we prove that when privacy-budgets are the same, feature-based partitioning requires fewer samples than sample-based one, and thus likely has better empirical performance. As transfer learning is difficult to be integrated with a differential privacy guarantee, we further combine the proposed method with hypothesis transfer learning to address the problem of learning across different organizations. Finally, we not only demonstrate the effectiveness of our method on two benchmark data sets, i.e., MNIST and NEWS20, but also apply it into a real application of cross-organizational diabetes prediction from RUIJIN data set, where privacy is of significant concern.

Motivation & Objective

• To address the performance degradation in privacy-preserving machine learning caused by noise injection in differential privacy.
• To enhance differentially private logistic regression using ensemble stacking, improving prediction accuracy without compromising privacy.
• To compare sample-based and feature-based partitioning strategies in stacking under differential privacy constraints.
• To integrate hypothesis transfer learning with differential privacy for cross-organizational learning, particularly in medical data settings.
• To validate the method on benchmark datasets and a real-world diabetes prediction application using the RUIJIN dataset.

Proposed method

• Proposes a differentially private stacking (PST) framework that partitions training data either by samples or by features to train multiple low-level models.
• Uses a high-level logistic regression model to combine predictions from low-level models, ensuring end-to-end differential privacy.
• Introduces feature-based partitioning where features are grouped based on importance, reducing sample complexity compared to sample-based partitioning.
• Applies a privacy budget allocation strategy across source and target domains in transfer learning, with ε = 1.0 for both.
• Incorporates feature importance scores from domain experts to weight groupings in feature-based PST, improving model performance.
• Employs a noise injection mechanism calibrated to ε-differential privacy, ensuring theoretical privacy guarantees.

Experimental results

Research questions

• RQ1 Does feature-based partitioning in differentially private stacking lead to lower sample complexity than sample-based partitioning?
• RQ2 Can integrating feature importance into the partitioning process further improve prediction performance under differential privacy?
• RQ3 How does the proposed PST method compare to standard privacy-preserving logistic regression (PLR) and transfer learning baselines in cross-organizational learning?
• RQ4 Can the proposed method achieve state-of-the-art performance in real-world medical prediction while preserving patient privacy?
• RQ5 What is the impact of noise injection on model performance, and can stacking effectively mitigate this degradation?

Key findings

• Feature-based partitioning in PST requires fewer samples than sample-based partitioning, demonstrating lower sample complexity and better empirical performance under the same privacy budget.
• PST-F(W), which uses expert-provided feature importance, achieves the highest AUC on the NEWS20 dataset (0.932) compared to other methods.
• On the RUIJIN diabetes dataset, PST-F(W) outperforms PLR(target) and PPHTL, achieving the best prediction performance in cross-organizational learning.
• The proposed method improves AUC by up to 0.035 over standard PLR on benchmark datasets, demonstrating significant performance gains.
• Empirical results confirm that feature importance integration boosts performance, especially in low-data regimes and cross-organizational settings.
• The method maintains ε-differential privacy (ε = 1.0) across all experiments, ensuring strong privacy guarantees.