[Paper Review] EGSDE: Unpaired Image-to-Image Translation via Energy-Guided Stochastic Differential Equations

Min Zhao, Fan Bao+2 more|arXiv (Cornell University)|Jul 14, 2022

Cancer-related molecular mechanisms research70 citations

TL;DR

EGSDE uses an energy function trained on both source and target domains to guide a pretrained score-based diffusion model, achieving better realism and faithfulness in unpaired I2I translation. It provides a product-of-experts interpretation and shows SOTA realism without sacrificing faithfulness across AFHQ and CelebA-HQ tasks.

ABSTRACT

Score-based diffusion models (SBDMs) have achieved the SOTA FID results in unpaired image-to-image translation (I2I). However, we notice that existing methods totally ignore the training data in the source domain, leading to sub-optimal solutions for unpaired I2I. To this end, we propose energy-guided stochastic differential equations (EGSDE) that employs an energy function pretrained on both the source and target domains to guide the inference process of a pretrained SDE for realistic and faithful unpaired I2I. Building upon two feature extractors, we carefully design the energy function such that it encourages the transferred image to preserve the domain-independent features and discard domain-specific ones. Further, we provide an alternative explanation of the EGSDE as a product of experts, where each of the three experts (corresponding to the SDE and two feature extractors) solely contributes to faithfulness or realism. Empirically, we compare EGSDE to a large family of baselines on three widely-adopted unpaired I2I tasks under four metrics. EGSDE not only consistently outperforms existing SBDMs-based methods in almost all settings but also achieves the SOTA realism results without harming the faithful performance. Furthermore, EGSDE allows for flexible trade-offs between realism and faithfulness and we improve the realism results further (e.g., FID of 51.04 in Cat to Dog and FID of 50.43 in Wild to Dog on AFHQ) by tuning hyper-parameters. The code is available at https://github.com/ML-GSAI/EGSDE.

Motivation & Objective

• Motivate leveraging source-domain training data in unpaired I2I using energy guidance.
• Design an energy function trained on both domains to balance realism and faithfulness.
• Preserve domain-independent features while discarding domain-specific ones during translation.
• Provide a theoretical interpretation as a product of experts for the EGSDE framework.
• Empirically validate on standard unpaired I2I benchmarks and show performance gains.

Proposed method

• Formulate conditional distribution p(y0|x0) via a reverse-time SDE that integrates an energy term into the pretrained SDE (Eq. 6).
• Define E(y,x,t) as a sum of two log-potentials E = lambda_s E_s + lambda_i E_i (Eq. 7).
• E_s uses a time-dependent domain-specific feature extractor to maximize similarity with the source (Eq. 9).
• E_i uses a domain-independent low-pass feature extractor to penalize differences with the source (Eq. 10).
• Solve the energy-guided reverse-time SDE with Euler-Maruyama updates incorporating grad_y E (Eq. 12).
• Offer a product-of-experts interpretation showing p(y_t|x0) ≈ p_r1(y_t|x0) p_e(y_t|x0) / Z_t (Eq. 13) and relate to a three-expert system (Eq. 16).

Experimental results

Research questions

• RQ1 Can an energy function pretrained on both source and target domains improve realism and faithfulness in unpaired I2I beyond target-domain diffusion alone?
• RQ2 How should the energy function be constructed to promote preservation of domain-independent features while discarding domain-specific ones?
• RQ3 Does the proposed EGSDE framework outperform SBDMs-based and GAN-based unpaired I2I baselines across standard tasks?
• RQ4 Does the method allow controllable trade-offs between realism and faithfulness?
• RQ5 Is there a theoretical interpretation of EGSDE as a product of experts that clarifies the roles of the SDE and the energy components?

Key findings

• EGSDE consistently outperforms SBDMs-based baselines on realism and faithfulness across Cat→Dog, Wild→Dog, and Male→Female tasks.
• EGSDE achieves state-of-the-art realism on AFHQ comparisons without harming faithfulness.
• Tuning hyper-parameters (λs, λi) and initial time M yields controllable realism-faithfulness trade-offs, improving realism metrics.
• Compared to SDEdit, EGSDE shows substantial FID improvements on the AFHQ benchmarks (examples in table).
• Human evaluation (AMT) indicates EGSDE is preferred over baselines in pairwise comparisons.
• The approach yields lower FID scores and competitive PSNR/SSIM, demonstrating strong overall image quality.