[Paper Review] The relativistic discriminator: a key element missing from standard GAN

Alexia Jolicoeur‐Martineau|arXiv (Cornell University)|Jul 2, 2018

Generative Adversarial Networks and Image Synthesis18 references580 citations

TL;DR

The paper introduces relativistic discriminators (RGANs and RaGANs) that compare real and fake data together, improving stability and sample quality across GAN variants and loss functions, with strong empirical gains on CIFAR-10 and CAT datasets.

ABSTRACT

In standard generative adversarial network (SGAN), the discriminator estimates the probability that the input data is real. The generator is trained to increase the probability that fake data is real. We argue that it should also simultaneously decrease the probability that real data is real because 1) this would account for a priori knowledge that half of the data in the mini-batch is fake, 2) this would be observed with divergence minimization, and 3) in optimal settings, SGAN would be equivalent to integral probability metric (IPM) GANs. We show that this property can be induced by using a relativistic discriminator which estimate the probability that the given real data is more realistic than a randomly sampled fake data. We also present a variant in which the discriminator estimate the probability that the given real data is more realistic than fake data, on average. We generalize both approaches to non-standard GAN loss functions and we refer to them respectively as Relativistic GANs (RGANs) and Relativistic average GANs (RaGANs). We show that IPM-based GANs are a subset of RGANs which use the identity function. Empirically, we observe that 1) RGANs and RaGANs are significantly more stable and generate higher quality data samples than their non-relativistic counterparts, 2) Standard RaGAN with gradient penalty generate data of better quality than WGAN-GP while only requiring a single discriminator update per generator update (reducing the time taken for reaching the state-of-the-art by 400%), and 3) RaGANs are able to generate plausible high resolutions images (256x256) from a very small sample (N=2011), while GAN and LSGAN cannot; these images are of significantly better quality than the ones generated by WGAN-GP and SGAN with spectral normalization.

Motivation & Objective

• Motivate why standard GAN discriminators miss a key property related to mini-batch composition (half real, half fake).
• Propose relativistic discriminator concepts that compare real vs. fake data jointly to better align with divergence minimization.
• Generalize relativistic ideas to non-standard GAN losses and define Relativistic GANs (RGANs) and Relativistic average GANs (RaGANs).
• Demonstrate empirical benefits in stability and data quality across datasets and architectures.

Proposed method

• Define a relativistic discriminator D that outputs the probability that a real sample is more realistic than a randomly sampled fake sample, via D( x_r, x_f ) = sigmoid(C(x_r) - C(x_f)).
• Derive Relativistic Standard GAN (RSGAN) losses where D and G losses depend on differences between real and fake critic values.
• Extend to Relativistic GANs (RGANs) with general loss forms F and G, showing IPM-based GANs are a subset when using the identity; introduce non-saturating versions.
• Introduce Relativistic average GANs (RaGANs) that compare data to the batch average of the opposing type, yielding RaSGAN, RaLSGAN, RaHingeGAN, and their GAN variants.
• Provide training algorithms (Algorithm 1 and Algorithm 2) for non-saturating RGANs and RaGANs.
• Evaluate on CIFAR-10 and CAT with standard architectures, using Fréchet Inception Distance (FID) as primary metric.

Experimental results

Research questions

• RQ1 Does incorporating a relativistic comparison between real and fake samples improve GAN stability and sample quality?
• RQ2 How do RGANS and RaGANs perform across standard and non-standard GAN losses compared to traditional SGAN, LSGAN, and WGAN variants?
• RQ3 Can relativistic discriminators enable high-quality generation at higher resolutions or with smaller datasets?
• RQ4 What is the impact of relativistic averaging (RaGAN) versus pointwise relativistic discrimination (RGAN) on training dynamics?
• RQ5 Do relativistic approaches reduce training instability under challenging settings (e.g., limited data, high resolutions)?

Key findings

• RGANs and RaGANs generally yield better stability and higher-quality samples than their non-relativistic counterparts.
• Relativistic SGAN (RSGAN) with gradient penalty (RSGAN-GP) achieves state-of-the-art-like FID on CIFAR-10 with a single discriminator update per generator update (FID 25.60).
• On CIFAR-10, RaLSGAN and RaSGAN improve over their non-relativistic versions in most stable setups; RaSGAN-GP shows strong performance as well.
• On the challenging CAT dataset, RaGANs produce lower minimum and mean FID and lower SD across 64x64, 128x128, and 256x256 resolutions; SGAN and LSGAN fail to converge at higher resolutions.
• RaGANs often outperform spectral normalization and gradient-penalty baselines in very difficult high-resolution CAT generation tasks.