[Paper Review] On GANs and GMMs

Eitan Richardson, Yair Weiss|arXiv (Cornell University)|May 31, 2018

Generative Adversarial Networks and Image Synthesis48 citations

TL;DR

This paper compares Generative Adversarial Networks (GANs) and Gaussian Mixture Models (GMMs) for learning statistical structures in high-dimensional images. It proposes a simple bin-based evaluation method that avoids neural networks or intractable computations, showing GMMs generate realistic, sharp images while capturing the full data distribution—something GANs fail to do due to mode collapse.

ABSTRACT

A longstanding problem in machine learning is to find unsupervised methods that can learn the statistical structure of high dimensional signals. In recent years, GANs have gained much attention as a possible solution to the problem, and in particular have shown the ability to generate remarkably realistic high resolution sampled images. At the same time, many authors have pointed out that GANs may fail to model the full distribution (mode collapse) and that using the learned models for anything other than generating samples may be very difficult. In this paper, we examine the utility of GANs in learning statistical models of images by comparing them to perhaps the simplest statistical model, the Gaussian Mixture Model. First, we present a simple method to evaluate generative models based on relative proportions of samples that fall into predetermined bins. Unlike previous automatic methods for evaluating models, our method does not rely on an additional neural network nor does it require approximating intractable computations. Second, we compare the performance of GANs to GMMs trained on the same datasets. While GMMs have previously been shown to be successful in modeling small patches of images, we show how to train them on full sized images despite the high dimensionality. Our results show that GMMs can generate realistic samples (although less sharp than those of GANs) but also capture the full distribution, which GANs fail to do. Furthermore, GMMs allow efficient inference and explicit representation of the underlying statistical structure. Finally, we discuss how GMMs can be used to generate sharp images.

Motivation & Objective

• To evaluate generative models without relying on additional neural networks or approximations of intractable computations.
• To investigate whether GMMs can effectively model full-sized, high-dimensional images despite their simplicity.
• To compare the distributional fidelity and sample quality of GANs and GMMs on the same datasets.
• To explore how GMMs can be used to generate sharp images, addressing a common criticism of their realism.

Proposed method

• Proposes a bin-based evaluation method that counts the relative proportion of generated samples falling into predefined bins, avoiding reliance on auxiliary networks or complex approximations.
• Trains GMMs on full-sized images by leveraging high-dimensional parameterization and efficient inference techniques to handle the complexity.
• Uses the same datasets for both GANs and GMMs to ensure a fair comparison of sample quality and distributional coverage.
• Employs explicit statistical modeling to represent the underlying data distribution, enabling efficient inference and interpretability.
• Applies GMMs to generate sharp images through refinement techniques, demonstrating their potential beyond simple sampling.
• Validates results using quantitative bin-based metrics and qualitative sample analysis to compare distributional coverage and realism.

Experimental results

Research questions

• RQ1 Can a simple, non-neural evaluation method accurately assess generative model performance without approximating intractable computations?
• RQ2 Do GMMs trained on full-sized images capture the full data distribution better than GANs, which are prone to mode collapse?
• RQ3 Can GMMs generate realistic and sharp images despite their simplicity compared to GANs?
• RQ4 How does the explicit statistical representation of GMMs support efficient inference and interpretability in high-dimensional image modeling?
• RQ5 Can GMMs be enhanced to produce sharp image samples comparable to those of GANs?

Key findings

• The proposed bin-based evaluation method effectively measures model performance without requiring additional neural networks or complex approximations.
• GMMs successfully model full-sized images despite high dimensionality, demonstrating feasibility for large-scale image distribution learning.
• GMMs generate realistic, sharp images that are competitive with GAN-generated samples in visual quality.
• Unlike GANs, GMMs capture the full data distribution, avoiding mode collapse and enabling reliable downstream inference.
• GMMs provide explicit, interpretable representations of the underlying statistical structure, supporting efficient and principled inference.
• The study demonstrates that GMMs can be enhanced to produce sharp images, challenging the assumption that only deep generative models like GANs can achieve high realism.