[Paper Review] The Blessing of Randomness: SDE Beats ODE in General Diffusion-based Image Editing

Shen Nie, Hanzhong Allan Guo+4 more|arXiv (Cornell University)|Nov 2, 2023

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes a unified probabilistic framework for diffusion-based image editing using stochastic differential equations (SDEs), demonstrating that SDEs consistently outperform ODEs in editing fidelity and alignment. By formulating editing as a latent variable manipulation via SDEs rather than ODEs, the method achieves superior performance across tasks like inpainting, image translation, and point-based dragging, with SDE-Drag showing significant gains over baselines including DragGAN and DiffEdit-ODE on a new benchmark (DragBench).

ABSTRACT

We present a unified probabilistic formulation for diffusion-based image editing, where a latent variable is edited in a task-specific manner and generally deviates from the corresponding marginal distribution induced by the original stochastic or ordinary differential equation (SDE or ODE). Instead, it defines a corresponding SDE or ODE for editing. In the formulation, we prove that the Kullback-Leibler divergence between the marginal distributions of the two SDEs gradually decreases while that for the ODEs remains as the time approaches zero, which shows the promise of SDE in image editing. Inspired by it, we provide the SDE counterparts for widely used ODE baselines in various tasks including inpainting and image-to-image translation, where SDE shows a consistent and substantial improvement. Moreover, we propose SDE-Drag -- a simple yet effective method built upon the SDE formulation for point-based content dragging. We build a challenging benchmark (termed DragBench) with open-set natural, art, and AI-generated images for evaluation. A user study on DragBench indicates that SDE-Drag significantly outperforms our ODE baseline, existing diffusion-based methods, and the renowned DragGAN. Our results demonstrate the superiority and versatility of SDE in image editing and push the boundary of diffusion-based editing methods.

Motivation & Objective

• To address the lack of a unified probabilistic framework for understanding diffusion-based image editing from a theoretical perspective.
• To investigate why SDE-based methods may outperform ODE-based counterparts in image editing despite similar empirical performance.
• To develop a practical, effective SDE-based method for point-based content dragging that avoids iterative optimization in latent space.
• To evaluate the generalization and robustness of SDE-based editing on open-set images, including real, artistic, and AI-generated content.
• To establish a new benchmark (DragBench) for evaluating image editing methods under realistic, open-set conditions.

Proposed method

• Proposes a unified probabilistic formulation where editing is modeled as a task-specific latent variable manipulation, followed by a new SDE or ODE for sampling, distinct from the original model's SDE/ODE.
• Theoretically proves that the Kullback-Leibler (KL) divergence between marginal distributions of SDEs decreases as time approaches zero, unlike ODEs, which provides a theoretical basis for SDE superiority.
• Develops SDE-Drag, a simple method that copies and pastes content points directly in latent space via SDE dynamics, avoiding optimization over multiple steps.
• Introduces a new hyperparameter β to control the strength of editing, with β=0.3 found optimal for dragging and β=1 for copying.
• Employs a multi-step SDE sampling process with a learned drift function to guide the latent variable toward the desired edit while preserving image quality.
• Constructs DragBench, a challenging benchmark with 100 open-set images (natural, artistic, and AI-generated) for evaluating editing methods under diverse, unseen conditions.

Experimental results

Research questions

• RQ1 Why do SDE-based image editing methods outperform ODE-based methods in practice, despite both being derived from the same pre-trained diffusion model?
• RQ2 Can a unified probabilistic formulation be established to generalize across diverse diffusion-based editing tasks such as inpainting, image-to-image translation, and point-based dragging?
• RQ3 Does the theoretical advantage of SDEs—specifically, the decreasing KL divergence over time—translate into measurable improvements in image editing quality and alignment?
• RQ4 Can a simple, non-optimization-based SDE method (SDE-Drag) achieve state-of-the-art performance in point-based content dragging compared to existing optimization-heavy or GAN-based approaches?
• RQ5 How well do SDE-based methods generalize to open-set images, including those from advanced diffusion models like Stable Diffusion and DALL·E 3?

Key findings

• SDE-Drag achieves significantly better user preference scores than the ODE baseline, existing diffusion-based methods (e.g., DiffEdit-ODE), and DragGAN on the DragBench benchmark.
• On the DiffEdit task, SDE-Drag achieves a 7.20 FID score at t₀=0.3, outperforming the ODE baseline (7.67 FID), with consistent improvements across all t₀ values.
• In image-to-image translation, DDIB-SDE achieves higher image fidelity than DDIB-ODE, as shown in visual comparisons and quantitative metrics.
• SDE-Drag successfully edits images from DALL·E 3 and Stable Diffusion, even when the original prompt is missing (e.g., dragging lightning into a mug), demonstrating strong prompt-image alignment.
• The method generalizes well to open-set images, including real photos, artworks, and AI-generated content, though failure cases exist for highly out-of-distribution content.
• The ablation study confirms β=0.3 as optimal for dragging and β=1 for copying, with stable performance across multiple random seeds.