[Paper Review] Convergence of score-based generative modeling for general data distributions

Holden Lee, Jianfeng Lu+1 more|arXiv (Cornell University)|Sep 26, 2022

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper establishes polynomial-time convergence guarantees for score-based generative modeling (SGM) on general data distributions, including multimodal and non-smooth ones, by analyzing denoising diffusion models under $L^2$-accurate score estimates. It provides Wasserstein and total variation distance bounds with no strong assumptions on smoothness or functional inequalities, achieving efficient, polynomial dependence on error parameters.

ABSTRACT

Score-based generative modeling (SGM) has grown to be a hugely successful method for learning to generate samples from complex data distributions such as that of images and audio. It is based on evolving an SDE that transforms white noise into a sample from the learned distribution, using estimates of the score function, or gradient log-pdf. Previous convergence analyses for these methods have suffered either from strong assumptions on the data distribution or exponential dependencies, and hence fail to give efficient guarantees for the multimodal and non-smooth distributions that arise in practice and for which good empirical performance is observed. We consider a popular kind of SGM -- denoising diffusion models -- and give polynomial convergence guarantees for general data distributions, with no assumptions related to functional inequalities or smoothness. Assuming $L^2$-accurate score estimates, we obtain Wasserstein distance guarantees for any distribution of bounded support or sufficiently decaying tails, as well as TV guarantees for distributions with further smoothness assumptions.

Motivation & Objective

• To close the gap between strong empirical performance and weak theoretical understanding of score-based generative models (SGM) in realistic settings.
• To provide convergence guarantees for SGM that are efficient (polynomial) and applicable to general data distributions, including multimodal and non-smooth ones.
• To analyze denoising diffusion models under $L^2$-accurate score estimates, avoiding the restrictive $L^∞$-error assumptions common in prior work.
• To derive Wasserstein and total variation distance bounds with minimal structural assumptions on the data distribution.
• To achieve polynomial dependence on error parameters, avoiding exponential dependencies seen in prior works.

Proposed method

• Analyzes the reverse-time SDE in denoising diffusion models, replacing the true score function with an $L^2$-accurate estimate.
• Uses a medium-time analysis to handle the non-uniform impact of $L^2$-error on the stationary distribution, unlike prior $L^\infty$-based analyses.
• Applies coupling arguments and Gaussian tail bounds to relate TV distance to Wasserstein error, especially for truncated samples.
• Employs a truncation step to handle heavy-tailed or unbounded distributions by replacing samples outside a ball with zero.
• Derives bounds on the required score estimation error $\varepsilon_\sigma$ to achieve target error $\varepsilon_{\text{TV}}$ or $\varepsilon_{\text{W}}$ in total variation or Wasserstein distance.
• Leverages assumptions on subexponential tails and bounded support to control tail behavior and derive polynomial dependencies.

Experimental results

Research questions

• RQ1 Can we achieve polynomial-time convergence guarantees for score-based generative modeling without assuming smoothness or functional inequalities on the data distribution?
• RQ2 How does $L^2$-accurate score estimation affect the convergence of the generated distribution to the data distribution in terms of Wasserstein and TV distance?
• RQ3 What is the required level of score estimation accuracy ($\varepsilon_\sigma$) to achieve a target distributional error ($\varepsilon_{\text{TV}}$ or $\varepsilon_{\text{W}}$)?
• RQ4 Can we handle distributions with bounded support or subexponential tails without requiring strong regularity conditions?
• RQ5 How does the truncation of samples outside a ball affect the Wasserstein error in the presence of heavy tails?

Key findings

• For distributions with bounded support or subexponential tails, the paper achieves $\operatorname{TV}(p_{t_N}, P_{\text{data}}) \leq \varepsilon_{\text{TV}}$ with $\varepsilon_\sigma = \widetilde{o}\left(\frac{\varepsilon_{\text{TV}}^{6.5}\varepsilon_{\text{W}}^5}{R^9 d^{2.25}}\right)$, showing polynomial dependence on error and dimension.
• Under additional smoothness assumptions, the required score error is $\varepsilon_\sigma = \widetilde{o}\left(\frac{\varepsilon_{\text{TV}}^4}{C^2 d}\right)$, improving over exponential dependencies in prior work.
• For Wasserstein error, the paper derives $\varepsilon_\sigma = \widetilde{o}\left(\frac{\varepsilon_{\text{W}}^{18}}{R^{22} d^{2.25}}\right)$ under general conditions, and $\varepsilon_\sigma = \widetilde{o}\left(\frac{\varepsilon_{\text{W}}^8}{C^2 R^8 d}\right)$ under stronger smoothness.
• The analysis avoids exponential dependence on diameter or inverse error, unlike prior works, by using a medium-time approach and careful coupling techniques.
• The results hold for multimodal and low-dimensional manifold-supported distributions, which violate standard assumptions like log-Sobolev inequalities.
• By truncating samples outside a ball and using Gaussian tail bounds, the method achieves Wasserstein error bounds even for unbounded distributions.