[Paper Review] Image Harmonization with Diffusion Model

Jiajie Li, Jian Wang+2 more|arXiv (Cornell University)|Jun 17, 2023

Advanced Image Processing TechniquesComputer Science3 citations

TL;DR

This paper proposes a novel image harmonization method using conditional diffusion models, leveraging both classifier-guided and classifier-free denoising diffusion frameworks to improve illumination and color consistency between foreground and background images. By training on latent space with ControlNet fine-tuning and integrating a color transfer mechanism, the approach achieves state-of-the-art performance in generating realistic, harmonized composites with improved appearance consistency and multi-output flexibility.

ABSTRACT

Image composition in image editing involves merging a foreground image with a background image to create a composite. Inconsistent lighting conditions between the foreground and background often result in unrealistic composites. Image harmonization addresses this challenge by adjusting illumination and color to achieve visually appealing and consistent outputs. In this paper, we present a novel approach for image harmonization by leveraging diffusion models. We conduct a comparative analysis of two conditional diffusion models, namely Classifier-Guidance and Classifier-Free. Our focus is on addressing the challenge of adjusting illumination and color in foreground images to create visually appealing outputs that seamlessly blend with the background. Through this research, we establish a solid groundwork for future investigations in the realm of diffusion model-based image harmonization.

Motivation & Objective

• To address the challenge of inconsistent lighting and color in image composites, where foreground and background images do not visually align.
• To explore the application of diffusion models—specifically DDPM and LDM—for image harmonization, overcoming limitations of traditional and deep learning-based methods.
• To improve appearance consistency in generated harmonized images by integrating classifier-guidance and color transfer techniques.
• To enable multi-output generation through the stochastic nature of diffusion models, offering users diverse, high-quality harmonization options.
• To validate the method on both synthetic and real-world composite datasets, demonstrating robustness and generalization beyond idealized training conditions.

Proposed method

• The method employs a conditional diffusion model framework using both classifier-guided and classifier-free denoising diffusion processes to generate harmonized images from unharmonized inputs.
• It leverages Latent Diffusion Models (LDM) with a pre-trained Stable Diffusion model, fine-tuned via ControlNet to preserve spatial and semantic structure during image editing.
• A color transfer mechanism is introduced to selectively transfer color statistics from the generated image to maintain appearance consistency while preserving foreground details.
• The approach uses brightness prediction to integrate background lighting cues, enhancing realism and consistency in illumination.
• The model is trained end-to-end on latent space, reducing computational cost while maintaining high-fidelity output generation.
• Multi-output generation is enabled by the inherent stochasticity of the diffusion process, allowing diverse results from the same input.

Experimental results

Research questions

• RQ1 Can conditional diffusion models effectively improve color and illumination harmony in image composites compared to traditional and deep learning-based methods?
• RQ2 How do classifier-guided and classifier-free diffusion models compare in terms of appearance consistency and realism in image harmonization?
• RQ3 To what extent can color transfer and lighting integration techniques enhance the visual quality and consistency of harmonized outputs?
• RQ4 Can the stochastic nature of diffusion models provide practical advantages in image harmonization, such as multiple diverse output options?
• RQ5 How well does the method generalize to real-world composite images when trained on synthetic datasets with idealized input conditions?

Key findings

• The proposed diffusion model-based approach achieves state-of-the-art performance on the iHarmony4 and HCOCO datasets, significantly outperforming existing methods in both quantitative metrics and visual quality.
• The classifier-free LDM with ControlNet fine-tuning produces high-fidelity harmonized images with improved structural and semantic consistency.
• The integration of color transfer significantly enhances detail preservation, as demonstrated by clearer text and object features in results (e.g., readable characters on gloves).
• Multi-output generation via the diffusion model’s stochasticity provides users with multiple visually plausible harmonization options, increasing flexibility in creative applications.
• The method generalizes well to real-world composites, as shown in comparisons with CDTNet on synthesized images from Open Images and Flick datasets, where it outperforms SOTA in lightness matching and object integration.
• Despite lower scores on conventional metrics, the diffusion model generates more realistic and perceptually consistent results, validating its superiority in perceptual quality.