[Paper Review] Loss-Sensitive Generative Adversarial Networks on Lipschitz Densities

Guo-Jun Qi|arXiv (Cornell University)|Jan 23, 2017

Generative Adversarial Networks and Image Synthesis30 references140 citations

TL;DR

Introduces Loss-Sensitive GAN (LS-GAN) with Lipschitz-regularized data density, proves distributional consistency and generalization, and extends to Generalized LS-GAN (GLS-GAN) and Conditional LS-GAN (CLS-GAN).

ABSTRACT

In this paper, we present the Lipschitz regularization theory and algorithms for a novel Loss-Sensitive Generative Adversarial Network (LS-GAN). Specifically, it trains a loss function to distinguish between real and fake samples by designated margins, while learning a generator alternately to produce realistic samples by minimizing their losses. The LS-GAN further regularizes its loss function with a Lipschitz regularity condition on the density of real data, yielding a regularized model that can better generalize to produce new data from a reasonable number of training examples than the classic GAN. We will further present a Generalized LS-GAN (GLS-GAN) and show it contains a large family of regularized GAN models, including both LS-GAN and Wasserstein GAN, as its special cases. Compared with the other GAN models, we will conduct experiments to show both LS-GAN and GLS-GAN exhibit competitive ability in generating new images in terms of the Minimum Reconstruction Error (MRE) assessed on a separate test set. We further extend the LS-GAN to a conditional form for supervised and semi-supervised learning problems, and demonstrate its outstanding performance on image classification tasks.

Motivation & Objective

• Motivate regularization of GANs by imposing a Lipschitz condition on data densities to improve generalization.
• Learn a loss function that ranks real samples lower than generated ones with margins.
• Develop a training objective that alternates loss-function learning and generator optimization to equilibrium.
• Show that LS-GAN converges to real data density as a regularization parameter grows.
• Extend the framework to generalized and conditional variants for broader applicability.

Proposed method

• Define a loss function L_theta and a generator G_phi with a margin-based constraint: L_theta(x) <= L_theta(G_phi(z)) - Delta(x, G_phi(z)).
• Relax the constraint with nonnegative slack xi and optimize jointly: min_theta,xi E_{x~P_data}[L_theta(x)] + lambda E_{x,z}[xi_x,z] subject to L_theta(x) - xi_x,z <= L_theta(G_phi(z)) - Delta(x,G_phi(z)).
• Form the LS-GAN objective S(theta, phi) = E_{x~P_data}[L_theta(x)] + lambda E_{x,z}[(Delta(x,G_phi(z)) + L_theta(x) - L_theta(G_phi(z)))_+].
• Define a secondary objective T(theta, phi) = E_{z~P_z}[L_theta(G_phi(z))] to optimize the generator.
• Prove distributional consistency: as lambda -> infinity, P_G* converges to P_data under Lipschitz data densities (Assumption 1).
• Introduce Generalized LS-GAN (GLS-GAN) with a cost function C(a) satisfying C(a) >= a and C(a)=a for a>=0; show LS-GAN and WGAN are special cases of GLS-GAN.

Experimental results

Research questions

• RQ1 Does LS-GAN with Lipschitz-regularized data densities ensure generated data matches real data density?
• RQ2 How does the LS-GAN performance compare to Wasserstein GAN and other regularized GANs in terms of generalization and sample quality?
• RQ3 Can LS-GAN be extended to generalized (GLS-GAN) and conditional (CLS-GAN) forms for supervised/semi-supervised tasks?
• RQ4 What are the generalization bounds and sample complexities for LS-GAN under Lipschitz assumptions?
• RQ5 Does bounding Lipschitz constants via gradient penalties improve training stability and generalization?

Key findings

• Under Lipschitz density regularity, LS-GAN generates samples whose density converges to the real data density as lambda grows.
• A Nash equilibrium exists with Lipschitz loss and generated density, ensuring distributional consistency (Theorem 1).
• Generalization bounds are provided: finite-sample S_m and T_k converge to their oracle counterparts with polynomial dependence on model size and Lipschitz constants (Theorems 2 and 3).
• GLS-GAN encompasses LS-GAN and WGAN as special cases, offering a family of regularized GANs via different cost functions (C a).
• Gradient penalties are proposed to bound Lipschitz constants, reducing sample complexity and improving stability (Section 5.2).
• CLS-GAN demonstrates competitive performance on image classification tasks, illustrating supervised/semi-supervised extensions (Section 8).