[Paper Review] Progressive Distillation for Fast Sampling of Diffusion Models

Tim Salimans, Jonathan Ho|arXiv (Cornell University)|Feb 1, 2022

Generative Adversarial Networks and Image Synthesis190 citations

TL;DR

The paper introduces progressive distillation to compress diffusion-model sampling, halving the number of steps iteratively to achieve high-quality samples with as few as 4 steps, while preserving performance.

ABSTRACT

Diffusion models have recently shown great promise for generative modeling, outperforming GANs on perceptual quality and autoregressive models at density estimation. A remaining downside is their slow sampling time: generating high quality samples takes many hundreds or thousands of model evaluations. Here we make two contributions to help eliminate this downside: First, we present new parameterizations of diffusion models that provide increased stability when using few sampling steps. Second, we present a method to distill a trained deterministic diffusion sampler, using many steps, into a new diffusion model that takes half as many sampling steps. We then keep progressively applying this distillation procedure to our model, halving the number of required sampling steps each time. On standard image generation benchmarks like CIFAR-10, ImageNet, and LSUN, we start out with state-of-the-art samplers taking as many as 8192 steps, and are able to distill down to models taking as few as 4 steps without losing much perceptual quality; achieving, for example, a FID of 3.0 on CIFAR-10 in 4 steps. Finally, we show that the full progressive distillation procedure does not take more time than it takes to train the original model, thus representing an efficient solution for generative modeling using diffusion at both train and test time.

Motivation & Objective

• Motivate and address the slow sampling bottleneck of diffusion models in unconditional and class-conditional image generation.
• Propose a distillation method that transfers accuracy from a slow, multi-step teacher to a faster student model.
• Demonstrate that progressive distillation preserves sample quality across standard benchmarks (CIFAR-10, ImageNet, LSUN) while reducing steps.
• Provide stable diffusion parameterizations and loss-weighting strategies that support rapid distillation without degradation.
• Show that the overall distillation process is computationally efficient relative to training the original model.

Proposed method

• Formulate progressive distillation that trains a student model to match two teacher DDIM steps with one student step, then iteratively halves the number of steps.
• Use a distillation target derived from running two DDIM steps with the teacher starting at a given z_t and inverting to obtain a sharp target for the student.
• Experiment with different parameterizations of the diffusion denoising model (direct x, epsilon, or v) and with loss weightings that remain informative at low signal-to-noise ratios.
• Train with a cosine noise schedule and a U-Net architecture with BigGAN-style up/downsampling, following standard diffusion-model training procedures.
• Compare distilled models against DDIM and stochastic baselines, and report FID (and IS where applicable) across datasets.
• Provide open-source implementation and detailed reproducibility notes.

Experimental results

Research questions

• RQ1 Can progressive distillation meaningfully reduce the number of sampling steps in diffusion models without substantial loss of sample quality?
• RQ2 What parameterizations and loss-weighting strategies yield stable and effective distillation across unconditional and class-conditional generation?
• RQ3 How does the quality of distilled models compare to existing fast-sampling methods on standard benchmarks (CIFAR-10, ImageNet, LSUN)?
• RQ4 What is the computational cost of progressive distillation relative to training the original diffusion model, and is it practically efficient?

Key findings

• Distilled models can generate high-quality samples with as few as 4 sampling steps on CIFAR-10, achieving competitive FID scores.
• Progressive distillation yields substantial speedups by halving the step count at each iteration, with the full procedure not exceeding the training time of the original model.
• Different stable parameterizations (direct x, epsilon, or v) and loss_weightings (SNR-based variants) produce robust performance, with one unstable combination identified in ablations.
• On CIFAR-10, 64×64 ImageNet, and 128×128 LSUN benchmarks, distilled models outperform or rival faster baselines at similar or lower step counts, particularly in the 4- to 8-step regime.
• The distillation process remains efficient: the total time does not exceed the time needed to train the original model, making test-time efficiency gains practical.
• Distilled models can also be used with stochastic samplers, yielding intermediate performance between DDIM and stochastic baselines.