[Paper Review] ArtFusion: Controllable Arbitrary Style Transfer using Dual Conditional Latent Diffusion Models

Dar-Yen Chen|arXiv (Cornell University)|Jun 15, 2023

Generative Adversarial Networks and Image Synthesis6 citations

TL;DR

ArtFusion proposes a diffusion-based arbitrary style transfer model using Dual Conditional Latent Diffusion Models (Dual-cLDM) that enables controllable, high-fidelity stylization by treating both content and style as conditions during training and inference. It achieves superior artistic detail preservation and avoids bias from similarity losses through classifier-free guidance and a content refiner, outperforming prior feed-forward methods in stylization quality and user-adjustable content-style balance.

ABSTRACT

Arbitrary Style Transfer (AST) aims to transform images by adopting the style from any selected artwork. Nonetheless, the need to accommodate diverse and subjective user preferences poses a significant challenge. While some users wish to preserve distinct content structures, others might favor a more pronounced stylization. Despite advances in feed-forward AST methods, their limited customizability hinders their practical application. We propose a new approach, ArtFusion, which provides a flexible balance between content and style. In contrast to traditional methods reliant on biased similarity losses, ArtFusion utilizes our innovative Dual Conditional Latent Diffusion Probabilistic Models (Dual-cLDM). This approach mitigates repetitive patterns and enhances subtle artistic aspects like brush strokes and genre-specific features. Despite the promising results of conditional diffusion probabilistic models (cDM) in various generative tasks, their introduction to style transfer is challenging due to the requirement for paired training data. ArtFusion successfully navigates this issue, offering more practical and controllable stylization. A key element of our approach involves using a single image for both content and style during model training, all the while maintaining effective stylization during inference. ArtFusion outperforms existing approaches on outstanding controllability and faithful presentation of artistic details, providing evidence of its superior style transfer capabilities. Furthermore, the Dual-cLDM utilized in ArtFusion carries the potential for a variety of complex multi-condition generative tasks, thus greatly broadening the impact of our research.

Motivation & Objective

• To address the limitations of feed-forward arbitrary style transfer (AST) models, which often produce rigid, over- or under-stylized results due to biased similarity losses.
• To overcome the paired data requirement in training conditional diffusion models for style transfer, which is typically unavailable in real-world scenarios.
• To enable flexible, user-controllable style transfer that balances content preservation and stylization intensity according to subjective preferences.
• To preserve fine-grained artistic details such as brushstrokes and genre-specific features without introducing repetitive artifacts.
• To demonstrate that diffusion models can be effectively adapted for AST without requiring paired training data or heavy optimization.

Proposed method

• ArtFusion employs a Dual Conditional Latent Diffusion Model (Dual-cLDM) that conditions the denoising process on both content and style features simultaneously, using a shared latent space.
• During training, the model is optimized via likelihood learning on a self-reconstruction task, where a single image serves as both content and style reference, eliminating the need for paired data.
• A content refiner is introduced to compress the content feature map, reducing its depth dimension from 16 to 12, to minimize residual style information and prevent over-reliance on content features.
• Classifier-free guidance (CFG) is extended into two dimensions (2D-CFG), allowing independent control over content and style guidance scales during inference for dynamic stylization adjustment.
• Style features are extracted using a pretrained VGG network, while content features are encoded via a first-stage VAE, and both are injected via adaptive normalization layers at each denoising step.
• Noise interpolation in the latent space enables smooth, continuous transitions between multiple artistic styles, facilitating hybrid style generation.

Figure 2: The inference framework of the Dual-cLDM for style transfer. Initiated from isotropic Gaussian-distributed noise $\hat{z}_{T}$ , the dual conditional backbone progressively denoises using both content and style as conditions. Post-denoising, the $\hat{z}_{0}$ is decoded using the first-sta

Experimental results

Research questions

• RQ1 Can a conditional diffusion model be effectively trained for arbitrary style transfer without paired training data?
• RQ2 How can content and style be balanced in a controllable manner during inference to reflect user preferences?
• RQ3 Can diffusion models avoid the bias of similarity-based losses (e.g., VGG mean/variance) that lead to artifacts and color distortion?
• RQ4 To what extent can diffusion models preserve subtle artistic details like brushwork and genre-specific features?
• RQ5 Can the dual conditional latent diffusion framework be generalized to other multi-condition generation tasks beyond style transfer?

Key findings

• ArtFusion outperforms existing feed-forward AST models in preserving fine artistic details such as brushstrokes and genre-specific characteristics, as confirmed by qualitative and quantitative evaluation.
• The model achieves superior stylization fidelity, with outputs that closely match the style reference in visual appearance, even when the reference is complex or abstract.
• The use of 2D-CFG enables dynamic, user-adjustable control over the content-style tradeoff, allowing users to range from under-stylization to over-stylization via guidance scale tuning.
• Ablation studies show that compressing the content feature from 16 to 12 dimensions achieves optimal balance, preventing both over-reliance on content features and loss of content structure.
• The model avoids common artifacts such as repetition and color distortion, which are prevalent in models relying on VGG-based style similarity losses.
• Interpolation in the latent space enables smooth, continuous blending between artistic styles, including hybrid styles formed by combining content and style references.

Figure 3: Left: Dual-cLDM Architecture. Pretrained VGG extracts style features $f_{s}$ , while content features $z_{c}$ are encoded using the first-stage VAE encoder. The content refiner processes $z_{c}$ into $z_{r}$ , refining content from inherent style. The refined $z_{r}$ is then concatenated w