[Paper Review] Theoretical limitations of Encoder-Decoder GAN architectures

Sanjeev Arora, Andrej Risteski+1 more|arXiv (Cornell University)|Nov 7, 2017

Generative Adversarial Networks and Image Synthesis4 references15 citations

TL;DR

This paper demonstrates that encoder-decoder GAN architectures—designed to mitigate mode collapse and learn meaningful features—can achieve near-optimal training objectives even when the generator produces a low-support distribution (severe mode collapse) and the encoder maps images to essentially white noise. The theoretical analysis reveals that the training objective alone cannot prevent these pathological solutions, implying that practical success must stem from inductive biases or optimization dynamics not captured by the objective function.

ABSTRACT

Encoder-decoder GANs architectures (e.g., BiGAN and ALI) seek to add an inference mechanism to the GANs setup, consisting of a small encoder deep net that maps data-points to their succinct encodings. The intuition is that being forced to train an encoder alongside the usual generator forces the system to learn meaningful mappings from the code to the data-point and vice-versa, which should improve the learning of the target distribution and ameliorate mode-collapse. It should also yield meaningful codes that are useful as features for downstream tasks. The current paper shows rigorously that even on real-life distributions of images, the encode-decoder GAN training objectives (a) cannot prevent mode collapse; i.e. the objective can be near-optimal even when the generated distribution has low and finite support (b) cannot prevent learning meaningless codes for data -- essentially white noise. Thus if encoder-decoder GANs do indeed work then it must be due to reasons as yet not understood, since the training objective can be low even for meaningless solutions.

Motivation & Objective

• To investigate whether encoder-decoder GANs, such as BiGAN and ALI, can theoretically prevent mode collapse and learn meaningful latent representations.
• To determine if the training objective of these architectures genuinely enforces distributional fidelity and semantically meaningful features.
• To analyze whether the joint generator-encoder objective provides theoretical guarantees against degenerate solutions, such as finite-support generators and random encodings.
• To extend prior theoretical results on standard GANs to the more complex encoder-decoder setting, revealing whether such architectures avoid the same limitations.

Proposed method

• Construct a theoretical framework where the generator is trained to produce a finite-support distribution over images, while the encoder maps real images to random noise via a partitioned, non-colliding set of latent codes.
• Define a discriminator that operates on paired (image, latent code) inputs and use a concentration argument via McDiarmid’s inequality to show that the expected discriminator score remains close to optimal under the constructed distribution.
• Use an epsilon-net over discriminators and a union bound to ensure the result holds uniformly across all possible discriminators of bounded size.
• Demonstrate that the expected value of the discriminator’s output under the real data distribution matches that under the generator’s output distribution, even when the generator has low support and the encoder produces essentially random codes.
• Leverage the fact that non-colliding sets of latent codes can be sampled independently from a uniform distribution over blocks, enabling concentration of measure arguments.
• Show that the training objective remains near-optimal even when the generator’s output is supported on only O(p log p / ε²) images, where p is the discriminator size.

Experimental results

Research questions

• RQ1 Can encoder-decoder GANs prevent mode collapse under realistic image distributions, despite their joint training objective?
• RQ2 Is the training objective of encoder-decoder GANs sufficient to enforce meaningful, non-random latent representations?
• RQ3 Do the theoretical guarantees of standard GANs extend to encoder-decoder architectures, or do they suffer from the same fundamental limitations?
• RQ4 Can a near-optimal training objective be achieved with a generator that has finite, small support and an encoder that outputs essentially white noise?
• RQ5 What theoretical mechanisms allow encoder-decoder GANs to work in practice if the objective function does not rule out degenerate solutions?

Key findings

• The training objective of encoder-decoder GANs can be near-optimal even when the generator’s output distribution has finite support of size O(p log p / ε²), indicating severe mode collapse is possible.
• The encoder can be constructed to map real images to white noise, producing a near-optimal solution under the training objective, demonstrating that meaningful feature learning is not enforced.
• The expected discriminator score under the real data distribution matches that under the generator’s distribution, even when the generator produces only a small number of distinct images.
• The concentration of measure argument, using McDiarmid’s inequality and a non-colliding set construction, ensures that the objective remains stable across random generator and latent code samples.
• The result holds uniformly across all discriminators of bounded size, achieved via an epsilon-net and union bound, extending the approach from Arora et al. (2017) to the encoder-decoder setting.
• The theoretical analysis shows that the training objective alone cannot prevent degenerate solutions, implying that practical success must rely on inductive biases or optimization dynamics not captured by the objective.