[Paper Review] SDEdit: Image Synthesis and Editing with Stochastic Differential Equations

Chenlin Meng, Yang Song+4 more|arXiv (Cornell University)|Aug 2, 2021

Generative Adversarial Networks and Image SynthesisComputer Science12 references80 citations

TL;DR

SDEdit proposes a novel image synthesis and editing framework using stochastic differential equations (SDEs) that enables flexible, zero-shot editing without task-specific loss functions or retraining. By denoising a noise-perturbed input through reverse SDE dynamics, it achieves high-quality image generation and editing guided by user inputs like strokes or composites, outperforming conditional GANs in adaptability and generalization.

ABSTRACT

We introduce a new image editing and synthesis framework, Stochastic Differential Editing (SDEdit), based on a recent generative model using stochastic differential equations (SDEs). Given an input image with user edits (e.g., hand-drawn color strokes), we first add noise to the input according to an SDE, and subsequently denoise it by simulating the reverse SDE to gradually increase its likelihood under the prior. Our method does not require task-specific loss function designs, which are critical components for recent image editing methods based on GAN inversion. Compared to conditional GANs, we do not need to collect new datasets of original and edited images for new applications. Therefore, our method can quickly adapt to various editing tasks at test time without re-training models. Our approach achieves strong performance on a wide range of applications, including image synthesis and editing guided by stroke paintings and image compositing.

Motivation & Objective

• To develop a flexible image editing framework that adapts to new editing tasks at test time without retraining.
• To eliminate the need for collecting paired datasets of original and edited images, which is required by conditional GANs.
• To avoid designing task-specific loss functions, a common bottleneck in recent GAN-based editing methods.
• To enable high-quality image synthesis and editing using only a pre-trained diffusion model and user-provided edits.
• To achieve strong performance across diverse editing tasks such as stroke-guided editing and image compositing.

Proposed method

• The method begins by adding noise to an input image using a forward SDE, transforming it into a diffusion process.
• It then applies reverse SDE dynamics to gradually denoise the image, increasing its likelihood under the learned data prior.
• User edits, such as color strokes or composited regions, are incorporated as conditioning signals during the reverse SDE denoising process.
• The framework leverages pre-trained score-based generative models, avoiding the need to fine-tune or retrain the underlying diffusion model.
• The reverse SDE is solved using numerical integration, allowing iterative refinement toward a high-quality, edited image.
• The approach is inherently generalizable, enabling rapid adaptation to new editing tasks at inference time.

Experimental results

Research questions

• RQ1 Can a single pre-trained diffusion model be used for diverse image editing tasks without retraining or fine-tuning?
• RQ2 How does SDEdit perform in comparison to conditional GANs that require paired datasets and task-specific loss functions?
• RQ3 To what extent can user edits such as hand-drawn strokes guide the image generation process in a zero-shot manner?
• RQ4 Can SDEdit achieve high-fidelity image synthesis and editing across multiple applications like compositing and stroke editing?
• RQ5 What is the impact of using SDE-based denoising versus alternative optimization-based or GAN-based inversion methods?

Key findings

• SDEdit achieves strong performance on image editing tasks without requiring task-specific loss functions, reducing design complexity.
• The method enables zero-shot adaptation to new editing tasks at test time, eliminating the need for retraining or fine-tuning.
• It outperforms conditional GAN-based methods in generalization, as it does not rely on paired training data for each editing task.
• The framework successfully handles diverse editing inputs, including stroke painting and image compositing, with high visual fidelity.
• By leveraging reverse SDE dynamics, SDEdit produces high-quality outputs that align well with user edits while preserving structural and semantic consistency.