[Paper Review] Privacy-Preserving Feature Selection with Secure Multiparty Computation

Xiling Li, Rafael Dowsley+1 more|arXiv (Cornell University)|Feb 6, 2021

Privacy-Preserving Technologies in Data32 references20 citations

TL;DR

This paper presents the first secure multiparty computation (MPC)-based protocol for privacy-preserving feature selection using the filter method, specifically leveraging a secure Gini impurity scoring mechanism. The protocol enables multiple parties to jointly select the top-k features without revealing raw data or feature identities, improving classifier accuracy while maintaining privacy in semi-honest and malicious threat models, with runtimes ranging from seconds to an hour depending on data size and security settings.

ABSTRACT

Existing work on privacy-preserving machine learning with Secure Multiparty Computation (MPC) is almost exclusively focused on model training and on inference with trained models, thereby overlooking the important data pre-processing stage. In this work, we propose the first MPC based protocol for private feature selection based on the filter method, which is independent of model training, and can be used in combination with any MPC protocol to rank features. We propose an efficient feature scoring protocol based on Gini impurity to this end. To demonstrate the feasibility of our approach for practical data science, we perform experiments with the proposed MPC protocols for feature selection in a commonly used machine-learning-as-a-service configuration where computations are outsourced to multiple servers, with semi-honest and with malicious adversaries. Regarding effectiveness, we show that secure feature selection with the proposed protocols improves the accuracy of classifiers on a variety of real-world data sets, without leaking information about the feature values or even which features were selected. Regarding efficiency, we document runtimes ranging from several seconds to an hour for our protocols to finish, depending on the size of the data set and the security settings.

Motivation & Objective

• To address the gap in privacy-preserving machine learning (PPML) by extending secure computation to the data preprocessing phase, particularly feature selection.
• To design a secure, efficient MPC protocol that enables private feature ranking without exposing raw feature values or selection outcomes.
• To demonstrate feasibility and effectiveness of the proposed protocol in real-world data science scenarios with practical performance.
• To support integration with any downstream ML model by using a model-agnostic filter-based approach.
• To lay the foundation for future PPML protocols targeting other preprocessing tasks like hyperparameter tuning, outlier detection, and handling missing values.

Proposed method

• Proposes a secure feature scoring protocol based on Gini impurity using MPC, enabling private computation of feature relevance scores across distributed parties.
• Designs a multi-party computation (MPC) protocol, π_GINI-FS, for filter-based feature selection that operates in 3PC or 4PC honest-majority settings over Z_q with q=2^64.
• Employs a secure variant of the Gini impurity calculation, MS-GINI, to compute feature importance without revealing individual data points or feature values.
• Uses MP-SPDZ to implement and benchmark the protocol, ensuring correctness and security under both semi-honest and malicious adversary models.
• Structures the protocol into sub-protocols: π_MS-GINI for secure Gini computation and π_FILTER-FS for feature ranking and selection.
• Executes the protocol in a distributed setting with co-located Azure virtual machines, measuring both computation and communication overhead.

Experimental results

Research questions

• RQ1 Can secure multiparty computation be effectively applied to the private selection of features in machine learning, independent of model training?
• RQ2 How can Gini impurity be securely computed in a multi-party setting to enable private feature scoring without exposing raw data?
• RQ3 What is the performance overhead of private feature selection using MPC in realistic data science workloads?
• RQ4 Does private feature selection with MPC improve downstream model accuracy without compromising privacy?
• RQ5 How do security models (semi-honest vs. malicious) affect the efficiency and scalability of the protocol?

Key findings

• The proposed MPC-based feature selection protocol improves the accuracy of logistic regression models across multiple real-world datasets, including Cognitive Load Detection, LSVT Voice Rehabilitation, and Speed Dating.
• Feature selection using the secure MS-GINI method achieves accuracy comparable to traditional non-private feature selection techniques such as Pearson correlation and mutual information.
• Runtime for the protocol ranges from several seconds to approximately one hour, depending on data set size and security model, with active (malicious) settings taking longer than passive (semi-honest) ones.
• The dominant performance bottleneck is matrix multiplication in the Gini computation, which scales with the number of instances (m), features (p), and selected features (k).
• The protocol successfully preserves privacy: no information about raw feature values or which features were selected is leaked to any party.
• The implementation in MP-SPDZ confirms the feasibility of integrating private feature selection into a standard machine learning-as-a-service pipeline with multiple untrusted servers.