[Paper Review] Privacy-Preserving Diffusion Model Using Homomorphic Encryption

Y.-B. Chen, Qiben Yan|arXiv (Cornell University)|Mar 9, 2024

Chaos-based Image/Signal Encryption4 citations

TL;DR

This paper proposes HE-Diffusion, a privacy-preserving stable diffusion framework that uses homomorphic encryption (HE) to securely perform inference on encrypted data, focusing on the denoising phase. By combining partial encryption with a min-distortion method and sparse tensor representation, it achieves 500x speedup over baseline HE methods, reducing encrypted inference time to the minute level while maintaining output quality on par with plaintext models.

ABSTRACT

In this paper, we introduce a privacy-preserving stable diffusion framework leveraging homomorphic encryption, called HE-Diffusion, which primarily focuses on protecting the denoising phase of the diffusion process. HE-Diffusion is a tailored encryption framework specifically designed to align with the unique architecture of stable diffusion, ensuring both privacy and functionality. To address the inherent computational challenges, we propose a novel min-distortion method that enables efficient partial image encryption, significantly reducing the overhead without compromising the model's output quality. Furthermore, we adopt a sparse tensor representation to expedite computational operations, enhancing the overall efficiency of the privacy-preserving diffusion process. We successfully implement HE-based privacy-preserving stable diffusion inference. The experimental results show that HE-Diffusion achieves 500 times speedup compared with the baseline method, and reduces time cost of the homomorphically encrypted inference to the minute level. Both the performance and accuracy of the HE-Diffusion are on par with the plaintext counterpart. Our approach marks a significant step towards integrating advanced cryptographic techniques with state-of-the-art generative models, paving the way for privacy-preserving and efficient image generation in critical applications.

Motivation & Objective

• To address privacy risks in stable diffusion model inference, where user prompts and generated images may be exposed to untrusted servers.
• To enable secure, end-to-end encrypted inference using homomorphic encryption (HE) without compromising model accuracy or performance.
• To overcome the high computational overhead of HE in deep learning by optimizing the denoising phase, the most computationally intensive part of diffusion models.
• To design an efficient, practical framework that allows service providers to compute on encrypted data without accessing sensitive inputs or outputs.
• To demonstrate that HE can be effectively integrated into state-of-the-art generative models for real-world privacy-sensitive applications.

Proposed method

• Proposes HE-Diffusion, a tailored homomorphic encryption framework specifically designed for the stable diffusion architecture, focusing on the denoising phase.
• Introduces a novel min-distortion method for partial image encryption that reduces computational overhead while preserving model output quality.
• Employs sparse tensor representation in COO format to accelerate homomorphic computations and improve efficiency.
• Avoids most nonlinear activation functions to minimize approximation errors and maintain accuracy in encrypted inference.
• Uses a cost function-based sparsity strategy (HILL) to enhance efficiency, though not optimized for maximum sparsity.
• Implements the framework in Python using TenSeal, with potential for performance gains via C++ porting and further sparse computation optimizations.

Figure 1: Sampling process of diffusion model at iteration $i$ . For the model forward component, the inputs are intermediate image $x_{i}$ and the prompts. Denoise process receives the output of model forward (i.e., noisy image $e_{i}$ and noise information $\alpha_{i}$ ), and then generates a bett

Experimental results

Research questions

• RQ1 Can homomorphic encryption be effectively applied to the denoising phase of stable diffusion models to preserve user input and output privacy?
• RQ2 How can the high computational cost of homomorphic encryption be mitigated in complex diffusion models without sacrificing output quality?
• RQ3 To what extent can partial encryption and sparse tensor representations reduce the performance overhead of HE-based inference in generative models?
• RQ4 Is it possible to achieve near-plaintext performance in encrypted stable diffusion inference while ensuring end-to-end privacy?
• RQ5 How can the integration of advanced cryptography with state-of-the-art generative models be made practical for real-world deployment?

Key findings

• HE-Diffusion achieves a 500x speedup compared to baseline homomorphic encryption methods, reducing encrypted inference time to the minute level.
• The model's output quality and performance are on par with the plaintext stable diffusion model, demonstrating minimal degradation from encryption.
• The partial encryption strategy based on min-distortion significantly reduces computational overhead while preserving image fidelity.
• Sparse tensor representation in COO format accelerates homomorphic operations, contributing to the overall efficiency of the framework.
• The framework successfully enables secure inference where the server never accesses unencrypted prompts or generated images.
• The approach is specialized for the denoising phase of stable diffusion, limiting immediate transferability to other model architectures without re-optimization.

Figure 2: When revising in the latent space with certain distortion, the image changes as the figure shows. We randomly set elements to 0 in the latent space to generate images with different distortion from the original image, and then transform back to pixel space to check the changes. As the dist