[Paper Review] Learning to Incorporate Texture Saliency Adaptive Attention to Image Cartoonization

Xiang Gao, Yuqi Zhang+1 more|arXiv (Cornell University)|Aug 2, 2022

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes a novel end-to-end GAN-based image cartoonization method that enhances cartoon style transfer by introducing a patch-level adversarial learning branch focused on texture-saliency-aware local regions. Using a novel Cartoon-Texture-Saliency-Sampler (CTSS) module, the model dynamically identifies and attends to high-impact local patches—such as edges and fine textures—enabling more vivid and abstract cartoon outputs with significantly improved FID scores compared to state-of-the-art methods.

ABSTRACT

Image cartoonization is recently dominated by generative adversarial networks (GANs) from the perspective of unsupervised image-to-image translation, in which an inherent challenge is to precisely capture and sufficiently transfer characteristic cartoon styles (e.g., clear edges, smooth color shading, abstract fine structures, etc.). Existing advanced models try to enhance cartoonization effect by learning to promote edges adversarially, introducing style transfer loss, or learning to align style from multiple representation space. This paper demonstrates that more distinct and vivid cartoonization effect could be easily achieved with only basic adversarial loss. Observing that cartoon style is more evident in cartoon-texture-salient local image regions, we build a region-level adversarial learning branch in parallel with the normal image-level one, which constrains adversarial learning on cartoon-texture-salient local patches for better perceiving and transferring cartoon texture features. To this end, a novel cartoon-texture-saliency-sampler (CTSS) module is proposed to dynamically sample cartoon-texture-salient patches from training data. With extensive experiments, we demonstrate that texture saliency adaptive attention in adversarial learning, as a missing ingredient of related methods in image cartoonization, is of significant importance in facilitating and enhancing image cartoon stylization, especially for high-resolution input pictures.

Motivation & Objective

• Address the challenge of insufficient transfer of salient cartoon texture features (e.g., clear edges, abstract patterns) in existing GAN-based image cartoonization methods.
• Overcome the limitation of global adversarial loss, which tends to overlook sparse, high-frequency cartoon features due to dominance of global smoothness and color patterns.
• Develop a lightweight, end-to-end framework that enhances stylization without requiring pre-processing steps like edge smoothing or additional style loss terms.
• Demonstrate that adaptive attention to texture-salient local regions is critical for achieving vivid, high-fidelity cartoon effects, especially in high-resolution images.

Proposed method

• Proposes a dual-discriminator architecture: an image-level discriminator for global style consistency (smooth shading, tone) and a patch-level discriminator for local texture pattern learning.
• Introduces the Cartoon-Texture-Saliency-Sampler (CTSS) module to dynamically sample local image patches with high cartoon-texture saliency, based on gradient-based saliency maps.
• Applies adversarial loss only on CTSS-selected patches in the patch-level branch, focusing learning on regions with prominent cartoon features such as sharp edges and fine textures.
• Combines the patch-level and image-level adversarial losses in a unified training objective to balance global coherence and local stylization.
• Uses standard perceptual and identity-preserving losses (e.g., L1, perceptual) to maintain content fidelity between input and generated images.
• Employs a multi-scale training strategy to ensure robustness across different input resolutions, especially benefiting high-resolution inputs.

Experimental results

Research questions

• RQ1 Can patch-level adversarial learning focused on salient local regions improve cartoon style transfer beyond global adversarial loss?
• RQ2 Does adaptive sampling of texture-salient patches lead to more vivid and abstract cartoon outputs compared to uniform or random patch sampling?
• RQ3 Can a simple patch-level adversarial loss outperform complex style loss components (e.g., Gram loss, mean-variance loss) in capturing cartoon texture patterns?
• RQ4 Is the proposed CTSS module essential for performance gains, or can fixed patch sampling achieve similar results?
• RQ5 How does the proposed method compare in performance and visual quality to state-of-the-art cartoonization models on high-resolution inputs?

Key findings

• The full model achieves a Fréchet Inception Distance (FID) of 115.25 on the CSC dataset, significantly outperforming the baseline with only image-level loss (FID: 145.25).
• Removing the patch-level adversarial branch leads to a 29.98-point increase in FID (from 115.25 to 145.25), confirming its critical role in stylization.
• The CTSS module is essential: removing it and using uniform patch sampling results in a FID of 127.09, a 11.84-point degradation compared to the full model.
• The model achieves state-of-the-art performance on three benchmark datasets (CSC, DB, TWR), with FID scores of 124.66, 112.97, and 115.25 respectively.
• Qualitative results show superior edge clarity, smoother color shading, and more abstract, cartoon-like textures, especially in high-resolution images and complex scenes.
• The method achieves strong performance without requiring edge-smoothing pre-processing (unlike CartoonGAN) or additional style loss terms (unlike AnimeGAN), making it more elegant and efficient.