[Paper Review] Image Harmonization with Region-wise Contrastive Learning

Jingtang Liang, Chi‐Man Pun|arXiv (Cornell University)|May 27, 2022

Image Enhancement Techniques4 citations

TL;DR

This paper proposes a novel image harmonization framework using region-wise contrastive learning and external style fusion to improve foreground-background appearance consistency. By leveraging encoder-extracted background features as style references and applying a region-wise contrastive loss that aligns foreground and background style representations, the method achieves state-of-the-art performance with improved visual realism and robustness on benchmark and real-world datasets.

ABSTRACT

Image harmonization task aims at harmonizing different composite foreground regions according to specific background image. Previous methods would rather focus on improving the reconstruction ability of the generator by some internal enhancements such as attention, adaptive normalization and light adjustment, $etc.$. However, they pay less attention to discriminating the foreground and background appearance features within a restricted generator, which becomes a new challenge in image harmonization task. In this paper, we propose a novel image harmonization framework with external style fusion and region-wise contrastive learning scheme. For the external style fusion, we leverage the external background appearance from the encoder as the style reference to generate harmonized foreground in the decoder. This approach enhances the harmonization ability of the decoder by external background guidance. Moreover, for the contrastive learning scheme, we design a region-wise contrastive loss function for image harmonization task. Specifically, we first introduce a straight-forward samples generation method that selects negative samples from the output harmonized foreground region and selects positive samples from the ground-truth background region. Our method attempts to bring together corresponding positive and negative samples by maximizing the mutual information between the foreground and background styles, which desirably makes our harmonization network more robust to discriminate the foreground and background style features when harmonizing composite images. Extensive experiments on the benchmark datasets show that our method can achieve a clear improvement in harmonization quality and demonstrate the good generalization capability in real-scenario applications.

Motivation & Objective

• To address the limitation of existing image harmonization methods in distinguishing and aligning foreground and background appearance features.
• To improve harmonization quality by enhancing the generator's ability to transfer background style to the foreground without degrading structural or semantic content.
• To introduce a region-wise contrastive loss that explicitly models style consistency between foreground and background patches.
• To leverage stable, encoder-extracted background features as external style guidance for more accurate and consistent style transfer in the decoder.
• To achieve better generalization in real-world scenarios while maintaining low computational overhead.

Proposed method

• Proposes a region-wise contrastive loss function, $\mathcal{L}_{HCL}$, that treats harmonized foreground patches as negative samples and ground-truth background patches as positive samples.
• Introduces external style fusion by using features from the encoder’s backbone as stable style references to guide the decoder’s feature transformation via adaptive instance normalization.
• Employs a shared multi-layer perceptron (MLP) to embed both positive and negative patch features into a shared latent space for contrastive optimization.
• Uses a UNet-based generator with style fusion applied at each decoder block, where background appearance features are fused with foreground features to improve realism.
• Applies a pixel-wise reconstruction loss ($\mathcal{L}_{pixel}$) alongside the contrastive loss to preserve structural and semantic fidelity.
• Optimizes the model using a combination of $\mathcal{L}_{pixel}$, $\mathcal{L}_{HCL}$, and a standard GAN loss to ensure high-quality, realistic outputs.

Experimental results

Research questions

• RQ1 Can a region-wise contrastive learning scheme improve the discrimination and alignment of foreground and background style features in image harmonization?
• RQ2 Does using stable, encoder-extracted background features as external style references lead to better harmonization than relying solely on internal decoder features?
• RQ3 How does the number of sampled patches affect the performance and training efficiency of the contrastive loss?
• RQ4 Can the proposed method generalize effectively to real-world, non-synthetic image compositions?
• RQ5 How does the proposed method compare to state-of-the-art approaches in terms of quantitative metrics and visual quality?

Key findings

• The proposed method achieves state-of-the-art performance on benchmark datasets, with a PSNR of 37.50 and SSIM of 99.04 when using 256 sampled patches.
• Using 256 patches for contrastive learning yields the best trade-off between performance and training complexity, as higher numbers (e.g., 512 or 1024) degrade results due to increased feature complexity and computational cost.
• The ablation study confirms that the region-wise contrastive loss significantly improves harmonization quality beyond style fusion alone, as shown by visual and quantitative improvements in foreground-background consistency.
• External style fusion using encoder features leads to more accurate background detail preservation compared to relying on decoder-reconstructed features, as validated by feature map comparisons.
• The method demonstrates strong generalization capability on real-scenario datasets, indicating practical applicability beyond synthetic benchmarks.
• Removing the contrastive loss or replacing it with a standard $\mathcal{L}_1$ loss results in a noticeable drop in performance, confirming its critical role in style alignment.