[Paper Review] Private Speech Classification with Secure Multiparty Computation

Kyle Bittner, Martine De Cock+1 more|arXiv (Cornell University)|Jul 1, 2020

Speech and Audio Processing46 references4 citations

TL;DR

This paper presents the first privacy-preserving deep learning approach for speech classification using Secure Multiparty Computation (MPC), enabling Alice to classify her speech signal with Bob’s encrypted model without revealing data or model parameters. In semi-honest settings, a 3.5-second audio clip is classified in under 0.3 seconds with full accuracy, demonstrating practical feasibility for real-time applications like voice assistants.

ABSTRACT

Deep learning in audio signal processing, such as human voice audio signal classification, is a rich application area of machine learning. Legitimate use cases include voice authentication, gunfire detection, and emotion recognition. While there are clear advantages to automated human speech classification, application developers can gain knowledge beyond the professed scope from unprotected audio signal processing. In this paper we propose the first privacy-preserving solution for deep learning-based audio classification that is provably secure. Our approach, which is based on Secure Multiparty Computation, allows to classify a speech signal of one party (Alice) with a deep neural network of another party (Bob) without Bob ever seeing Alice's speech signal in an unencrypted manner. As threat models, we consider both passive security, i.e. with semi-honest parties who follow the instructions of the cryptographic protocols, as well as active security, i.e. with malicious parties who deviate from the protocols. We evaluate the efficiency-security-accuracy trade-off of the proposed solution in a use case for privacy-preserving emotion detection from speech with a convolutional neural network. In the semi-honest case we can classify a speech signal in under 0.3 sec; in the malicious case it takes $\sim$1.6 sec. In both cases there is no leakage of information, and we achieve classification accuracies that are the same as when computations are done on unencrypted data.

Motivation & Objective

• To address privacy leakage in deep learning-based speech classification, where unsecured processing can expose sensitive personal data like emotion, gender, or health status.
• To enable secure, private inference of speech signals using a third-party’s deep neural network without exposing raw audio or model weights.
• To evaluate the efficiency, security, and accuracy trade-offs of MPC-based speech classification under both passive and active threat models.
• To demonstrate that MPC can support real-time, high-accuracy speech classification suitable for production systems like digital assistants.

Proposed method

• The protocol uses Secure Multiparty Computation (MPC) to allow Alice and Bob to jointly compute the output of a 1D convolutional neural network (CNN) on Alice’s encrypted speech signal.
• All computations are performed over secret-shared values using the MP-SPDZ framework, ensuring that no party learns the raw input or model weights during inference.
• The system supports both 2PC (two-party computation) and 3PC (three-party computation) settings, with 3PC offering better performance and stronger security guarantees.
• The CNN architecture is specifically optimized for MPC efficiency through architectural choices and quantization to reduce computational overhead.
• For active security, the protocol includes checks to detect and prevent malicious deviations by either party, ensuring correctness even under adversarial behavior.
• The protocol reveals only the final class label (e.g., emotion class), which is shared only upon agreement, preserving end-to-end privacy.

Experimental results

Research questions

• RQ1 Can MPC enable private inference of speech signals using a third-party’s deep learning model without exposing the audio or model parameters?
• RQ2 What is the performance overhead of MPC-based speech classification in terms of runtime and accuracy compared to unencrypted inference?
• RQ3 How does the system perform under both passive (semi-honest) and active (malicious) security models?
• RQ4 Can the system achieve real-time inference for practical applications like voice assistants, given the computational cost of MPC?

Key findings

• In the semi-honest threat model, a 3.5-second speech signal is classified in 0.26 seconds with full accuracy, matching unencrypted inference.
• With lower-accuracy configuration, classification completes in 0.15 seconds, demonstrating high efficiency in MPC-optimized settings.
• In the malicious threat model, classification takes approximately 1.6 seconds, with no information leakage and full accuracy preserved.
• The system maintains classification accuracy identical to unencrypted inference, proving that privacy does not compromise performance.
• The 3PC setting with three semi-honest servers reduces runtime compared to 2PC, especially under active security, due to more efficient cryptographic operations.
• The results show that MPC-based private speech classification is feasible for real-time applications, such as digital assistants, when optimized for efficiency.