[Paper Review] BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrapping

Jiatao Gu, Shuangfei Zhai+3 more|arXiv (Cornell University)|Jun 8, 2023

Advanced Neuroimaging Techniques and Applications4 citations

TL;DR

BOOT proposes a data-free knowledge distillation method for denoising diffusion models that uses bootstrapping from consecutive sampling steps to train a single-step student model without real data or extensive offline data generation. It achieves comparable image quality to teacher models while enabling orders-of-magnitude faster inference, even for large-scale text-to-image models like Stable Diffusion and DeepFloyd IF.

ABSTRACT

Diffusion models have demonstrated excellent potential for generating diverse images. However, their performance often suffers from slow generation due to iterative denoising. Knowledge distillation has been recently proposed as a remedy that can reduce the number of inference steps to one or a few without significant quality degradation. However, existing distillation methods either require significant amounts of offline computation for generating synthetic training data from the teacher model or need to perform expensive online learning with the help of real data. In this work, we present a novel technique called BOOT, that overcomes these limitations with an efficient data-free distillation algorithm. The core idea is to learn a time-conditioned model that predicts the output of a pre-trained diffusion model teacher given any time step. Such a model can be efficiently trained based on bootstrapping from two consecutive sampled steps. Furthermore, our method can be easily adapted to large-scale text-to-image diffusion models, which are challenging for conventional methods given the fact that the training sets are often large and difficult to access. We demonstrate the effectiveness of our approach on several benchmark datasets in the DDIM setting, achieving comparable generation quality while being orders of magnitude faster than the diffusion teacher. The text-to-image results show that the proposed approach is able to handle highly complex distributions, shedding light on more efficient generative modeling.

Motivation & Objective

• To address the slow inference speed of diffusion models, which require iterative denoising over many steps.
• To overcome limitations of existing distillation methods that require large-scale synthetic data generation or access to real training data.
• To enable efficient distillation of large-scale text-to-image diffusion models where training data is inaccessible due to privacy or scale constraints.
• To develop a data-free, efficient training paradigm that leverages bootstrapping from consecutive time-step predictions.
• To achieve high-fidelity, single-step generation that matches the quality of full-diffusion teachers while drastically reducing inference time.

Proposed method

• The method trains a time-conditioned student model to predict outputs of a pre-trained diffusion teacher across all time steps, given a single noise input.
• It uses a novel Signal-ODE derived from the PF-ODE to stabilize training and improve generalization across time steps.
• Bootstrapping is enabled by training the model to predict $\bm{x}_t$ based on knowledge of $\bm{x}_{t'}$ for $t' > t$, leveraging the deterministic mapping along the same diffusion trajectory.
• A boundary loss is applied periodically during training to improve sampling quality and diversity, with reduced frequency to maintain efficiency.
• The approach is compatible with classifier-free guidance (CFG), allowing effective distillation of conditional and text-to-image models.
• The student model is trained end-to-end using only the teacher’s forward pass and no real data, making it suitable for privacy-sensitive or large-scale models.

Experimental results

Research questions

• RQ1 Can a single-step student model be trained effectively without access to real training data or large-scale synthetic data generation?
• RQ2 How can bootstrapping from consecutive time-step predictions improve distillation efficiency and performance in data-free settings?
• RQ3 Can the proposed method achieve high-fidelity image generation comparable to full-diffusion teachers while enabling orders-of-magnitude faster inference?
• RQ4 How well does the method generalize to complex, high-capacity text-to-image diffusion models like Stable Diffusion and DeepFloyd IF?
• RQ5 What role does the Signal-ODE and boundary loss play in stabilizing training and improving sampling diversity?

Key findings

• BOOT achieves generation quality comparable to the full-diffusion teacher model on unconditional and class-conditional image generation benchmarks, including FFHQ, LSUN, and ImageNet.
• The method enables single-step inference with inference speedups of 50–1000× compared to the teacher model, depending on the dataset and setup.
• On text-to-image generation, BOOT successfully distills both DeepFloyd IF and Stable Diffusion models, producing high-quality images from diverse prompts without access to training data.
• The distilled models maintain high diversity and fidelity, as evidenced by qualitative comparisons using the same initial noise across teacher and student.
• The boundary loss, applied every 4 training steps, improves sampling quality and diversity without significantly increasing training cost.
• The method is computationally efficient, requiring only 8 A100 GPUs and 3–7 days of training for models up to 1B parameters, with no data loading overhead beyond text prompts.