[Paper Review] Differentially Private Sliced Wasserstein Distance

Alain Rakotomamonjy, Liva Ralaivola|arXiv (Cornell University)|Jul 5, 2021

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper proposes a differentially private variant of the Sliced Wasserstein Distance (SWD) by applying a Gaussian mechanism to the random projection process inherent in SWD, resulting in a new distance called the Smoothed Sliced Wasserstein Distance (GS-WSD). The method ensures $(\varepsilon,\delta)$-differential privacy while preserving the metric and scalability properties of SWD, enabling effective privacy-preserving training of generative models and domain adaptation systems with minimal accuracy drop.

ABSTRACT

Developing machine learning methods that are privacy preserving is today a central topic of research, with huge practical impacts. Among the numerous ways to address privacy-preserving learning, we here take the perspective of computing the divergences between distributions under the Differential Privacy (DP) framework -- being able to compute divergences between distributions is pivotal for many machine learning problems, such as learning generative models or domain adaptation problems. Instead of resorting to the popular gradient-based sanitization method for DP, we tackle the problem at its roots by focusing on the Sliced Wasserstein Distance and seamlessly making it differentially private. Our main contribution is as follows: we analyze the property of adding a Gaussian perturbation to the intrinsic randomized mechanism of the Sliced Wasserstein Distance, and we establish the sensitivityof the resulting differentially private mechanism. One of our important findings is that this DP mechanism transforms the Sliced Wasserstein distance into another distance, that we call the Smoothed Sliced Wasserstein Distance. This new differentially private distribution distance can be plugged into generative models and domain adaptation algorithms in a transparent way, and we empirically show that it yields highly competitive performance compared with gradient-based DP approaches from the literature, with almost no loss in accuracy for the domain adaptation problems that we consider.

Motivation & Objective

• To develop a differentially private distribution distance that maintains the scalability and metric properties of the Sliced Wasserstein Distance (SWD).
• To analyze the privacy guarantees of applying a Gaussian mechanism to the random projection process in SWD.
• To demonstrate that the resulting mechanism yields a new distance, the Smoothed Sliced Wasserstein Distance, which inherits key metric properties.
• To enable seamless integration of this private distance into existing machine learning algorithms such as generative models and domain adaptation frameworks.
• To empirically validate the method on both low- and high-dimensional datasets, including $64 \times 64$ RGB CelebA, under strict privacy budgets.

Proposed method

• Apply the Gaussian mechanism to the random projection matrix $\mathbf{U}$ in the Sliced Wasserstein Distance, adding zero-mean Gaussian noise $\mathbf{V}$ to the projected data $\mathbf{X}\mathbf{U}$.
• Define the resulting differentially private mechanism as DP-SWD, which transforms the original SWD into a smoothed version, termed the Gaussian Smoothed Sliced Wasserstein Distance (GS-WSD).
• Establish privacy guarantees by bounding the sensitivity of the projected data under the Gaussian mechanism, using both the Central Limit Theorem (CLT) and Bernstein inequality approximations.
• Use the CLT-based bound for tighter sensitivity estimates and the Bernstein bound for more conservative, dimension-independent control.
• Plug the DP-SWD loss into generative adversarial networks (e.g., GANs) and domain adaptation models as a privacy-preserving alternative to standard divergence metrics like MMD or SWD.
• Calibrate noise variance $\sigma$ using the PyTorch privacy package to achieve a target privacy budget of $\varepsilon = 10$, $\delta = 10^{-5}$ or $10^{-6}$ on large-scale datasets.

Experimental results

Research questions

• RQ1 Can the Gaussian mechanism be applied to the random projection step of the Sliced Wasserstein Distance to achieve differential privacy without compromising utility?
• RQ2 What is the sensitivity of the resulting differentially private mechanism, and how can it be bounded in high-dimensional settings?
• RQ3 Does the resulting private distance retain the metric properties of a true distance, and can it be used as a drop-in replacement in existing ML algorithms?
• RQ4 How does the performance of DP-SWD compare to gradient-based DP methods in generative modeling and domain adaptation tasks?
• RQ5 Can DP-SWD be effectively scaled to high-dimensional, real-world datasets such as $64 \times 64$ RGB CelebA?

Key findings

• The DP-SWD mechanism achieves $(\varepsilon,\delta)$-differential privacy with $\varepsilon = 10$, $\delta = 10^{-5}$ on MNIST and FashionMNIST, with minimal accuracy loss in domain adaptation tasks.
• On MNIST and FashionMNIST, DP-SWD achieves test accuracy of 78% and 66% respectively with $\varepsilon=10$, $\delta=10^{-5}$, outperforming DP-CGAN and DP-MERF.
• For generative modeling on MNIST, DP-SWD achieves a Fréchet Inception Distance (FID) score of 97 using the Bernstein bound, compared to 158 with the CLT-based bound.
• On the $64 \times 64$ RGB CelebA dataset, DP-SWD achieves an FID score of 97 (vs. 158 with CLT) under $\varepsilon=10$, $\delta=10^{-6}$, marking the first successful DP generative model on this dataset.
• The proposed method enables privacy-preserving training of GANs with only ~180K parameters, achieving competitive performance in under 10 minutes, while GS-WGAN requires ~4M parameters and hours of training.
• The DP-SWD mechanism transforms SWD into a new distance, the Gaussian Smoothed Sliced Wasserstein Distance, which inherits metric properties and enables plug-and-play integration into existing ML pipelines.