[Paper Review] Differentially Private Latent Diffusion Models

M Liu, Saiyue Lyu+2 more|arXiv (Cornell University)|May 25, 2023

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper proposes DP-LDMs, a method that achieves state-of-the-art differentially private image generation by fine-tuning only attention modules and conditioning embedders in latent diffusion models (LDMs) using DP-SGD. By training in the latent space and reducing trainable parameters by ~90%, the approach generates high-quality 256×256 images with FID scores as low as 19.0 at ε=10, outperforming prior work on CIFAR10, CelebA64, and CelebAHQ.

ABSTRACT

Diffusion models (DMs) are one of the most widely used generative models for producing high quality images. However, a flurry of recent papers points out that DMs are least private forms of image generators, by extracting a significant number of near-identical replicas of training images from DMs. Existing privacy-enhancing techniques for DMs, unfortunately, do not provide a good privacy-utility tradeoff. In this paper, we aim to improve the current state of DMs with differential privacy (DP) by adopting the extit{Latent} Diffusion Models (LDMs). LDMs are equipped with powerful pre-trained autoencoders that map the high-dimensional pixels into lower-dimensional latent representations, in which DMs are trained, yielding a more efficient and fast training of DMs. Rather than fine-tuning the entire LDMs, we fine-tune only the $ extit{attention}$ modules of LDMs with DP-SGD, reducing the number of trainable parameters by roughly $90\%$ and achieving a better privacy-accuracy trade-off. Our approach allows us to generate realistic, high-dimensional images (256x256) conditioned on text prompts with DP guarantees, which, to the best of our knowledge, has not been attempted before. Our approach provides a promising direction for training more powerful, yet training-efficient differentially private DMs, producing high-quality DP images. Our code is available at https://anonymous.4open.science/r/DP-LDM-4525.

Motivation & Objective

• To address the challenge of training high-quality differentially private diffusion models (DMs) given the scalability issues of DP-SGD on large models.
• To improve upon existing fine-tuning approaches that train entire DMs under DP-SGD by reducing the number of trainable parameters.
• To enable high-dimensional (256×256) differentially private image generation, including text-to-image and class-conditional generation, which has not been achieved before.
• To democratize DP image generation by reducing computational cost and carbon footprint compared to full-model DP training.

Proposed method

• Pre-train the entire LDM, including the autoencoder and diffusion U-Net, on public data (e.g., LAION-400M or ImageNet).
• Fine-tune only the attention modules and conditioning embedders (e.g., BERT for text prompts) using DP-SGD on private data, drastically reducing the number of trainable parameters (~10% of full model).
• Utilize the latent space representation via a pre-trained autoencoder to reduce model complexity and accelerate training.
• Apply DP-SGD with carefully tuned noise scale and clipping to ensure (ε, δ)-differential privacy during fine-tuning.
• Freeze the BERT text encoder during fine-tuning to further reduce parameter count, then re-enable it during sampling.
• Perform ablation studies to evaluate the impact of parameter reduction and layer-specific fine-tuning on FID and privacy-utility trade-offs.

Experimental results

Research questions

• RQ1 Can fine-tuning only attention modules in LDMs achieve better privacy-utility trade-offs than full-model fine-tuning under DP-SGD?
• RQ2 Can DP-LDMs generate high-resolution (256×256) images with competitive FID scores while preserving privacy?
• RQ3 Does training in the latent space significantly improve training efficiency and model performance compared to pixel-space diffusion models under DP?
• RQ4 Can DP-LDMs support both text-to-image and class-conditional generation with strong fidelity at low privacy budgets (e.g., ε=1)?

Key findings

• DP-LDM achieves SOTA FID scores of 19.0 ± 0.0 on CelebAHQ 256×256 at ε=10, compared to 200.8 for DP-MEPF, demonstrating a significant improvement in image quality.
• At ε=1, DP-LDM achieves an FID of 25.6 ± 0.1 on CelebAHQ, outperforming prior methods that fail to generate plausible samples at this privacy level.
• The method reduces the number of trainable parameters by approximately 90% compared to full-model fine-tuning, enabling faster and more efficient training.
• DP-LDM generates high-fidelity, text-conditioned images at 256×256 resolution with FID 44.5 at ε=10 and 55.0 at ε=1, showing strong prompt adherence even under strict privacy constraints.
• Training time is reduced to just 10 GPU hours, compared to 192 GPU hours in prior work, significantly lowering computational and environmental costs.
• Ablation studies confirm that fine-tuning only attention modules and embedders yields better performance than fine-tuning the full model, validating the parameter-efficiency strategy.