[论文解读] On the Statistical Capacity of Deep Generative Models

Edric Tam, David B. Dunson|ArXiv.org|Jan 14, 2025

Bayesian Methods and Mixture Models被引用 3

一句话总结

论文表明，具有常见潜变量分布（如高斯）的深度生成模型并非通用生成器；其输出集中于轻尾，与重尾目标分布不符；给出维度无关的集中性结果及对流形与扩散模型的扩展。

ABSTRACT

Deep generative models are routinely used in generating samples from complex, high-dimensional distributions. Despite their apparent successes, their statistical properties are not well understood. A common assumption is that with enough training data and sufficiently large neural networks, deep generative model samples will have arbitrarily small errors in sampling from any continuous target distribution. We set up a unifying framework that debunks this belief. We demonstrate that broad classes of deep generative models, including variational autoencoders and generative adversarial networks, are not universal generators. Under the predominant case of Gaussian latent variables, these models can only generate concentrated samples that exhibit light tails. Using tools from concentration of measure and convex geometry, we give analogous results for more general log-concave and strongly log-concave latent variable distributions. We extend our results to diffusion models via a reduction argument. We use the Gromov--Levy inequality to give similar guarantees when the latent variables lie on manifolds with positive Ricci curvature. These results shed light on the limited capacity of common deep generative models to handle heavy tails. We illustrate the empirical relevance of our work with simulations and financial data.

研究动机与目标

• Motivate the study of statistical properties of deep generative models in sampling from implicit, high-dimensional targets.
• Establish that Gaussian and related latent variable choices lead to light-tailed outputs, challenging the universal approximation belief.
• Extend tail-concentration results to log-concave, strongly log-concave latent variables and latent variables on manifolds with positive Ricci curvature.
• Provide reductions to diffusion models and demonstrate dimension-free concentration guarantees under broad settings.

提出的方法

• Model latent-variable generation as x_i = f(z_i) + ε_i where z_i ~ P and ε_i ~ Q; focus on the Lipschitz properties of the fitted network ￢.
• Prove sub-Gaussian concentration for Gaussian latent variables: Pr(|u^T[￢(z)-E￢(z)￢]| ￢≥ t) ≤ 2 exp(-t^2/C_p^2).
• Generalize to log-concave latent variables with sub-exponential tails: Pr(|u^T[￢(z)-E￢(z)￢]| ≥ t) ≤ 2 exp(-t/C_p).
• Establish strong log-concavity results yielding sub-Gaussian tails with bounds depending on γ (strong log-concavity).
• Extend to latent-variable manifolds with positive Ricci curvature via Gromov-Levy inequality, yielding sub-Gaussian concentration for ￢∘φ(z).
• Provide a diffusion-model reduction showing Gaussian-latent variable diffusion steps yield sub-Gaussian tails for X_0 with constants depending on step-wise Lipschitz factors.

(a) Samples from bivariate Cauchy distribution centred at $0$ with identity scale matrix

实验结果

研究问题

• RQ1 Do common deep generative models (GANs, VAEs) universally approximate arbitrary continuous target distributions when latent variables are Gaussian (or similarly simple)?
• RQ2 What tail behavior do generative mappings ￢ have when latent variables follow Gaussian, log-concave, or strongly log-concave distributions?
• RQ3 How do latent-variable manifolds with positive Ricci curvature affect concentration properties of generated samples?
• RQ4 Can diffusion-based generative models be analyzed via Lipschitz concentration, and do they inherit light-tailed behavior from Gaussian latents?

主要发现

• For Gaussian latent variables, the centered output ￢(z)-E￢(z)￢ is sub-Gaussian along any unit direction, implying light tails.
• For log-concave and strongly log-concave latent variables, the centered output is sub-exponential or sub-Gaussian with dimension-free or poly-logarithmic dependence on dimension.
• Latent variables on manifolds with positive Ricci curvature yield sub-Gaussian concentration for the mapped outputs via a Lipschitz embedding.
• Diffusion models with Gaussian latent variables also yield sub-Gaussian tails for the generated sample X_0, through a reduction to a single Lipschitz transformation on augmented Gaussian vectors.
• Numerical simulations and financial data illustrate that GANs and diffusion models capture center mass well but underrepresent tail behavior, aligning with the theoretical light-tail guarantees.
• The results imply that many deep generative models are not universal generators for heavy-tailed target distributions and may underestimate uncertainty in applications like anomaly detection and finance.

(b) Samples from fitted generative adversarial network