[Paper Review] Adversarial Lipschitz Regularization

Dávid Terjék|Repository of the Academy's Library (Library of the Hungarian Academy of Sciences)|Jul 12, 2019

Generative Adversarial Networks and Image Synthesis32 references21 citations

TL;DR

This paper proposes Adversarial Lipschitz Regularization (ALR), a novel method that explicitly penalizes violations of the Lipschitz constraint in neural networks by generating adversarial perturbations via power iteration, enabling stable training of Wasserstein GANs. ALR achieves state-of-the-art Inception Score and competitive Fréchet Inception Distance on CIFAR-10 and effective performance in high-dimensional settings on CelebA-HQ, outperforming gradient penalty in batch-normalized critic architectures.

ABSTRACT

Generative adversarial networks (GANs) are one of the most popular approaches when it comes to training generative models, among which variants of Wasserstein GANs are considered superior to the standard GAN formulation in terms of learning stability and sample quality. However, Wasserstein GANs require the critic to be 1-Lipschitz, which is often enforced implicitly by penalizing the norm of its gradient, or by globally restricting its Lipschitz constant via weight normalization techniques. Training with a regularization term penalizing the violation of the Lipschitz constraint explicitly, instead of through the norm of the gradient, was found to be practically infeasible in most situations. Inspired by Virtual Adversarial Training, we propose a method called Adversarial Lipschitz Regularization, and show that using an explicit Lipschitz penalty is indeed viable and leads to competitive performance when applied to Wasserstein GANs, highlighting an important connection between Lipschitz regularization and adversarial training.

Motivation & Objective

• To address the instability and mode collapse in GAN training by enforcing strict Lipschitz continuity on the critic network.
• To develop a viable alternative to implicit gradient norm penalties (e.g., gradient penalty) by explicitly regularizing Lipschitz violations.
• To explore the connection between adversarial training and Lipschitz regularization, inspired by Virtual Adversarial Training (VAT).
• To evaluate the effectiveness of explicit Lipschitz regularization in both low- and high-dimensional generative modeling settings.

Proposed method

• ALR uses power iteration to compute an adversarial direction $ r_{\text{adv}} $ at each input point $ x $, approximating the steepest direction of change in the critic's output.
• The method computes $ r_{\text{adv}} = \arg\max_{\|r\|_2=1} \left| f(x + r) - f(x) \right| $ using iterative power iteration to estimate the directional derivative.
• An explicit Lipschitz penalty term is applied as $ \lambda \cdot \mathbb{E}_{x} \left( \left| \frac{f(x) - f(x + r_{\text{adv}})}{\|r_{\text{adv}}\|_2} - 1 \right|^2 \right) $, penalizing deviation from unit slope.
• The penalty is integrated into the WGAN objective, forming WGAN-ALP, which enforces 1-Lipschitz continuity directly rather than via gradient norm constraints.
• For high-dimensional settings, the method adapts the penalty to include both absolute and squared terms of the violation to improve stability.
• The approach is validated on CIFAR-10 and CelebA-HQ, comparing against standard gradient penalty and baseline WGANs.

Experimental results

Research questions

• RQ1 Can explicit regularization of Lipschitz violations via adversarial perturbations lead to stable and high-quality GAN training?
• RQ2 How does ALR compare to gradient penalty in terms of Inception Score and Fréchet Inception Distance on standard benchmarks?
• RQ3 Does ALR maintain performance when batch normalization is used in the critic, where gradient penalties are known to fail?
• RQ4 Can ALR be effectively applied in high-dimensional image generation tasks such as Progressive Growing GAN on CelebA-HQ?
• RQ5 What is the relationship between adversarial training and Lipschitz regularization, and can it be leveraged to improve model generalization?

Key findings

• WGAN-ALP achieved a state-of-the-art Inception Score of 9.41 on CIFAR-10 among non-progressive growing methods, outperforming baseline WGAN and WGAN-GP.
• On the CelebA-HQ dataset, ALP achieved a Fréchet Inception Distance (FID) of 14.65, compared to 8.69 for the original GP-based Progressive GAN, indicating strong performance despite not surpassing GP in this case.
• In the WGAN-BN setting, where gradient penalty failed (best Inception Score: 6.29), ALP achieved a significantly improved score of 8.71, demonstrating robustness to batch normalization.
• The method successfully generated high-quality samples on both CIFAR-10 and CelebA-HQ, with visual quality comparable to state-of-the-art models.
• ALR was found to be more effective than competing methods in the presence of batch normalization layers, suggesting a key advantage in practical architectures.
• The two-sided penalty formulation (19) was less stable than the one-sided version, indicating that one-sided regularization is more reliable in practice.