[Paper Review] AutoPrivacy: Automated Layer-wise Parameter Selection for Secure Neural Network Inference

Qian Lou, Song Bian+1 more|arXiv (Cornell University)|Jun 7, 2020

Cryptography and Data Security22 references17 citations

TL;DR

AutoPrivacy proposes a deep reinforcement learning-based system to automatically select homomorphic encryption (HE) parameters layer-wise in hybrid privacy-preserving neural networks (HPPNNs), significantly reducing inference latency. By tailoring HE parameters—plaintext modulus, ciphertext modulus, and polynomial degree—to each layer’s error tolerance, it achieves 53–70% lower latency than prior HPPNNs with negligible accuracy loss, enabling faster and more efficient secure inference on encrypted data.

ABSTRACT

Hybrid Privacy-Preserving Neural Network (HPPNN) implementing linear layers by Homomorphic Encryption (HE) and nonlinear layers by Garbled Circuit (GC) is one of the most promising secure solutions to emerging Machine Learning as a Service (MLaaS). Unfortunately, a HPPNN suffers from long inference latency, e.g., $\sim100$ seconds per image, which makes MLaaS unsatisfactory. Because HE-based linear layers of a HPPNN cost $93\%$ inference latency, it is critical to select a set of HE parameters to minimize computational overhead of linear layers. Prior HPPNNs over-pessimistically select huge HE parameters to maintain large noise budgets, since they use the same set of HE parameters for an entire network and ignore the error tolerance capability of a network. In this paper, for fast and accurate secure neural network inference, we propose an automated layer-wise parameter selector, AutoPrivacy, that leverages deep reinforcement learning to automatically determine a set of HE parameters for each linear layer in a HPPNN. The learning-based HE parameter selection policy outperforms conventional rule-based HE parameter selection policy. Compared to prior HPPNNs, AutoPrivacy-optimized HPPNNs reduce inference latency by $53\%\sim70\%$ with negligible loss of accuracy.

Motivation & Objective

• To address the high inference latency in hybrid privacy-preserving neural networks (HPPNNs), which stems from inefficient homomorphic encryption (HE) parameter selection.
• To overcome the limitations of prior HPPNNs that use uniform, over-pessimistic HE parameters across all layers, leading to excessive computational overhead.
• To enable automated, layer-specific HE parameter selection that respects each layer’s unique error tolerance and computational demands.
• To reduce inference latency and communication cost while maintaining high accuracy and 128-bit security in secure neural network inference.
• To provide a scalable, non-expert-friendly solution for deploying fast and accurate privacy-preserving machine learning in real-world MLaaS settings.

Proposed method

• AutoPrivacy employs a deep reinforcement learning agent to learn an optimal policy for selecting HE parameters (plaintext modulus p, ciphertext modulus q, polynomial degree n) for each linear layer in a HPPNN.
• The agent is trained using a reward function that balances inference latency, communication cost, and model accuracy, with penalties for decryption failures.
• The method models each layer’s error tolerance and noise accumulation characteristics to guide HE parameter selection, avoiding over-provisioning.
• It uses a fixed, compact neural network architecture (e.g., MobileNet) with uniform quantization, focusing optimization on layer-wise HE parameters rather than architecture or bitwidth.
• The system evaluates HE parameter choices via simulation of homomorphic operations and noise growth over computation paths.
• It leverages a curriculum learning strategy to gradually increase the complexity of the search space, improving training stability and convergence.

Experimental results

Research questions

• RQ1 Can a learning-based approach outperform rule-based HE parameter selection in reducing inference latency for HPPNNs?
• RQ2 Can layer-wise HE parameter selection, tailored to individual layer error tolerance, significantly reduce computational overhead without sacrificing accuracy?
• RQ3 How does automated HE parameter selection compare to joint architecture and quantization search in terms of efficiency and performance?
• RQ4 Can deep reinforcement learning effectively explore the high-dimensional HE parameter space to find near-optimal configurations for each layer?
• RQ5 To what extent can AutoPrivacy reduce latency and communication cost in secure neural network inference while maintaining security and accuracy?

Key findings

• AutoPrivacy reduces HPPNN inference latency by 53–70% compared to prior HPPNNs, with the best-performing configuration cutting latency from ~100 seconds to ~30 seconds per CIFAR-10 image.
• The system achieves a 79% reduction in communication overhead (from 2GB to 0.4GB) compared to DELPHI, which uses the same network and dataset.
• AutoPrivacy maintains 91.4% top-1 accuracy on CIFAR-10 using MobileNet with 14-bit quantization, matching or exceeding prior results with significantly lower latency.
• The search process takes only 8 hours, compared to over 60 hours for NASS, which performs joint architecture and quantization search, demonstrating superior search efficiency.
• AutoPrivacy identifies that depth-wise convolution layers can safely use smaller n and q due to fewer accumulations and less noise growth, while point-wise layers require larger parameters to tolerate higher noise.
• The method discovers that residual block layers (e.g., 4th and 8th) require larger noise budgets than others, justifying higher HE parameter settings for accuracy preservation.