[Paper Review] Artist Style Transfer Via Quadratic Potential

Rahul Bhalley, Jianlin Su|arXiv (Cornell University)|Feb 14, 2019

Generative Adversarial Networks and Image Synthesis31 references4 citations

TL;DR

This paper proposes CycleGAN-QP, a novel artist style transfer method using quadratic potential divergence (QP) to stabilize GAN training without Lipschitz constraints, while integrating identity and cycle-consistency losses and replacing transpose convolutions with upsampling to eliminate checkerboard artifacts. The method achieves high-quality, artifact-free stylized images across multiple artistic styles, including Van Gogh, Monet, Cézanne, and Ukiyo-e.

ABSTRACT

In this paper we address the problem of artist style transfer where the painting style of a given artist is applied on a real world photograph. We train our neural networks in adversarial setting via recently introduced quadratic potential divergence for stable learning process. To further improve the quality of generated artist stylized images we also integrate some of the recently introduced deep learning techniques in our method. To our best knowledge this is the first attempt towards artist style transfer via quadratic potential divergence. We provide some stylized image samples in the supplementary material. The source code for experimentation was written in PyTorch and is available online in my GitHub repository.

Motivation & Objective

• Address the instability and training difficulties in GAN-based image-to-image translation, particularly for artist style transfer.
• Improve training stability and convergence in unsupervised image translation by replacing traditional divergences with quadratic potential divergence (QP).
• Eliminate checkerboard artifacts in generated stylized images by replacing transpose convolutions with nearest-neighbor upsampling followed by convolution.
• Achieve high-fidelity style transfer and reconstruction between unpaired real and artistic image domains using cycle-consistency and identity loss.
• Demonstrate the effectiveness of QP-GAN in generating realistic, high-quality stylized images across diverse artistic styles.

Proposed method

• Adopt quadratic potential divergence (QP) as the training objective for the critic network, enabling stable GAN training without requiring 1-Lipschitz constraints on the critic.
• Formulate the critic loss using QP divergence, which avoids gradient vanishing and simplifies training compared to WGAN or standard GANs.
• Integrate cycle-consistency and identity loss terms into the generator objective to preserve content and improve reconstruction fidelity.
• Replace transpose convolution layers in the generator with nearest-neighbor upsampling followed by convolution to suppress checkerboard artifacts.
• Train the generator and critic networks end-to-end using Adam optimizer with a fixed learning rate of 0.0002 and β1=0.5, β2=0.999.
• Use data augmentation via random horizontal flipping and center-cropping to 256×256 resolution, with normalization using mean and std of 0.5 per channel.

Experimental results

Research questions

• RQ1 Can quadratic potential divergence improve training stability and convergence in GAN-based artist style transfer without requiring critic weight clipping or spectral normalization?
• RQ2 How does the proposed QP-GAN framework compare to standard GANs and WGANs in terms of image quality and artifact suppression in style transfer?
• RQ3 To what extent does replacing transpose convolutions with upsampling+convolution reduce checkerboard artifacts in generated stylized images?
• RQ4 Can the combination of cycle-consistency and identity loss improve reconstruction quality and preserve content during style transfer?
• RQ5 Does the method generalize across diverse artistic styles such as Van Gogh, Monet, Cézanne, and Ukiyo-e with minimal hyperparameter tuning?

Key findings

• The proposed CycleGAN-QP method successfully generates high-quality stylized images across multiple artistic styles, including Van Gogh, Monet, Cézanne, and Ukiyo-e, as shown in qualitative samples.
• Checkerboard artifacts were completely eliminated in generated images by replacing transpose convolutions with nearest-neighbor upsampling followed by convolution.
• The method achieved stable training using quadratic potential divergence, avoiding the need for explicit critic weight constraints or gradient penalties.
• The model was trained for 15,000 iterations (12,000 for Ukiyo-e) on a single NVIDIA K80 GPU, taking approximately 1.25 days per training run.
• The use of identity and cycle-consistency losses improved reconstruction quality, enabling the network to recover original content images from stylized outputs.
• The method demonstrates state-of-the-art performance in unsupervised artist style transfer, with no reported mode collapse or training failure during experiments.