[論文レビュー] An Edit Friendly DDPM Noise Space: Inversion and Manipulations

Inbar Huberman-Spiegelglas, В. А. Куликов+1 more|arXiv (Cornell University)|Apr 12, 2023

Music and Audio Processing被引用数 7

ひとこと要約

The paper introduces an edit-friendly DDPM noise space with a fast inversion method that perfectly reconstructs images and enables diverse, text-guided edits without fine-tuning.

ABSTRACT

Denoising diffusion probabilistic models (DDPMs) employ a sequence of white Gaussian noise samples to generate an image. In analogy with GANs, those noise maps could be considered as the latent code associated with the generated image. However, this native noise space does not possess a convenient structure, and is thus challenging to work with in editing tasks. Here, we propose an alternative latent noise space for DDPM that enables a wide range of editing operations via simple means, and present an inversion method for extracting these edit-friendly noise maps for any given image (real or synthetically generated). As opposed to the native DDPM noise space, the edit-friendly noise maps do not have a standard normal distribution and are not statistically independent across timesteps. However, they allow perfect reconstruction of any desired image, and simple transformations on them translate into meaningful manipulations of the output image (e.g. shifting, color edits). Moreover, in text-conditional models, fixing those noise maps while changing the text prompt, modifies semantics while retaining structure. We illustrate how this property enables text-based editing of real images via the diverse DDPM sampling scheme (in contrast to the popular non-diverse DDIM inversion). We also show how it can be used within existing diffusion-based editing methods to improve their quality and diversity. Webpage: https://inbarhub.github.io/DDPM_inversion

研究の動機と目的

• Motivate robust editing of real images using DDPMs by addressing inversion challenges.
• Define an edit-friendly noise space that preserves image structure during editing.
• Develop a fast inversion procedure that yields perfect reconstruction without optimizing model parameters.
• Show how fixing edit-friendly noise maps enables reliable text-guided edits and diversity.

提案手法

• Introduce an alternative DDPM latent noise space where noise maps are highly correlated across timesteps and have higher variance, enabling edit-friendly manipulations.
• Propose Algorithm 1 for inversion that imputes x_t via x_t = sqrt(alpha_bar_t) x_0 + sqrt(1 - alpha_bar_t) tilde{epsilon}_t with independent tilde{epsilon}_t.
• Extract noise maps z_t from the inverted sequence using z_t = (x_{t-1} - hat{mu}_t(x_t)) / sigma_t and reconstruct x_{t-1} with the diffusion update to avoid error accumulation.
• Demonstrate that these noise maps, while not standard normal, allow perfect reconstruction and enable faithful structure preservation when editing.
• Show integration with existing editing methods (e.g., Prompt-to-Prompt, Zero-Shot I2I) by replacing DDIM inversions with the edit-friendly inversion.

Figure 2: The native and edit friendly noise spaces. When sampling an image using DDPM (left), there is access to the “ground truth” noise maps that generated it. This native noise space, however, is not edit friendly (2nd column). For example, fixing those noise maps and changing the text prompt, c

実験結果

リサーチクエスチョン

• RQ1 Can an inversion that uses edit-friendly noise maps improve fidelity and structure preservation for real-image editing under DDPMs?
• RQ2 Do correlated, higher-variance noise maps facilitate diverse, text-guided edits without fine-tuning or attention-map manipulation?
• RQ3 How does the edit-friendly inversion interact with existing diffusion-based editing methods to improve fidelity and diversity?

主な発見

Method	CLIP sim. ↑	LPIPS ↓	Diversity ↑	Time
DDIM inv.	0.31	0.62	0.00	39
PnP	0.31	0.36	0.00	206
P2P	0.30	0.61	0.00	40
P2P+Our	0.31	0.25	0.11	48
Our inv.	0.32	0.29	0.18	36

• The edit-friendly noise space yields higher-variance, negatively correlated noise across timesteps, embedding image structure more strongly.
• Inversion via Algorithm 1 reconstructs x0 exactly (up to numerical error) and enables diverse edited outputs when text prompts are varied.
• Fixing the edit-friendly noise maps while changing the text prompt preserves structure and artifacts are reduced compared to DDIM inversions.
• Integrating edit-friendly inversion into P2P and Zero-Shot I2I improves fidelity (structure/texture preservation) and maintains CLIP alignment.
• Compared to DDIM inversion and CycleDiffusion, the proposed method achieves a better balance of CLIP similarity, LPIPS, and diversity with faster edit times.
• The approach enables color edits, shifting, and text-guided edits with preserved textures and global structure.

Figure 3: The DDPM latent noise space. In DDPM, the generative (reverse) diffusion process synthesizes an image $x_{0}$ in $T$ steps, by utilizing $T+1$ noise maps, $\{x_{T},z_{T},\ldots,z_{1}\}$ . We regard those noise maps as the latent code associated with the generated image.