[Paper Review] SDDM: Score-Decomposed Diffusion Models on Manifolds for Unpaired Image-to-Image Translation

Shikun Sun, Longhui Wei+3 more|arXiv (Cornell University)|Aug 4, 2023

Mycobacterium research and diagnosis4 citations

TL;DR

SDDM proposes a score-decomposed diffusion model on manifolds for unpaired image-to-image translation, decomposing the score function into denoising and content refinement components via manifold-based optimization. By leveraging block adaptive instance normalization and multi-objective optimization, SDDM achieves state-of-the-art performance with significantly fewer diffusion steps than prior SBDM-based methods.

ABSTRACT

Recent score-based diffusion models (SBDMs) show promising results in unpaired image-to-image translation (I2I). However, existing methods, either energy-based or statistically-based, provide no explicit form of the interfered intermediate generative distributions. This work presents a new score-decomposed diffusion model (SDDM) on manifolds to explicitly optimize the tangled distributions during image generation. SDDM derives manifolds to make the distributions of adjacent time steps separable and decompose the score function or energy guidance into an image ``denoising" part and a content ``refinement" part. To refine the image in the same noise level, we equalize the refinement parts of the score function and energy guidance, which permits multi-objective optimization on the manifold. We also leverage the block adaptive instance normalization module to construct manifolds with lower dimensions but still concentrated with the perturbed reference image. SDDM outperforms existing SBDM-based methods with much fewer diffusion steps on several I2I benchmarks.

Motivation & Objective

• To address the lack of explicit control over intermediate generative distributions in existing score-based diffusion models (SBDMs) for unpaired image-to-image translation.
• To explicitly separate the denoising and content refinement components of the score function during image generation.
• To enable multi-objective optimization by equalizing the refinement parts of the score and energy guidance on a manifold.
• To construct lower-dimensional, concentrated manifolds using a block adaptive instance normalization (BAdaIN) module for better optimization efficiency.
• To improve generation quality and convergence speed with fewer diffusion steps compared to existing SBDM-based approaches.

Proposed method

• SDDM introduces a manifold-based optimization framework where the tangent space separates the score function into a denoising part and a content refinement part.
• The method decomposes both the score function and energy guidance into components that can be optimized in parallel, enabling multi-objective learning.
• Block adaptive instance normalization (BAdaIN) is used to construct low-dimensional manifolds that concentrate around the perturbed reference image, replacing traditional low-pass filters.
• The model enforces equal footing between the refinement parts of the score and energy guidance, allowing balanced gradient combination.
• An iterative refinement policy is applied, with increased iterations at a key middle time step $T_0$ to improve fidelity without excessive computation.
• The framework is validated through ablation studies on block count and $T_0$ selection, showing improved trade-offs between FID and SSIM.

Figure 1: The overview of our SDDM. At each time step, compared with directly adding energy guidance to the score function, we firstly use the moments of the distribution $\mathbf{y}_{t}$ as constraints to get the manifolds $\mathcal{M}_{t}$ at each time step $t$ . Then, we restrict potential energy

Experimental results

Research questions

• RQ1 Can the intermediate distributions in score-based diffusion models be explicitly optimized during image generation in unpaired image-to-image translation?
• RQ2 How can the score function be decomposed into distinct denoising and content refinement components for better control?
• RQ3 Can manifold-based optimization separate adjacent time-step distributions to reduce interference between guidance components?
• RQ4 What is the impact of using BAdaIN over traditional low-pass filters in constructing manifolds for conditional generation?
• RQ5 How does multi-objective optimization of score and energy guidance improve generation quality with fewer diffusion steps?

Key findings

• SDDM achieves the best FID of 62.29 and SSIM of 0.422 on the FF-Horse dataset, outperforming existing SBDM-based methods with fewer diffusion steps.
• Policy 3, which increases iterations at $T_0$, achieves the best balance between FID and SSIM, indicating that targeted refinement improves performance with minimal cost.
• Using $16\times16$ blocks yields the best trade-off between fidelity and detail, as $8\times8$ blocks lead to lower FID but worse SSIM, while $32\times32$ blocks degrade performance.
• The manifold constructed via BAdaIN is equivalently concentrated with the perturbed reference image, enabling effective decomposition and optimization.
• The proposed method reduces interference between guidance components by equalizing refinement parts, leading to more stable and controllable generation.
• SDDM demonstrates superior performance on two I2I benchmarks, confirming the effectiveness of score decomposition and manifold-based multi-objective optimization.

Figure 2: The mean and standard deviation of $L_{2}$ norms of $\mathbf{s}_{r}$ in $T_{\mathbf{x}_{t}}\mathcal{M}$ and other part in $N_{\mathbf{x}_{t}}\mathcal{M}$ . We repeat 100 times of our SDDM with different reference images.