[Paper Review] Improved ArtGAN for Conditional Synthesis of Natural Image and Artwork

Wei Tan, Chee Seng Chan+2 more|arXiv (Cornell University)|Aug 31, 2017

Generative Adversarial Networks and Image Synthesis7 references4 citations

TL;DR

This paper proposes ArtGAN, a conditional GAN that improves image synthesis by backpropagating label gradients to the generator and integrating a categorical autoencoder into the discriminator for better feature learning. The model achieves state-of-the-art Inception scores on CIFAR-10 and generates high-quality, realistic images and artworks across diverse styles, genres, and artists, including abstract and non-figurative representations.

ABSTRACT

This paper proposes a series of new approaches to improve Generative Adversarial Network (GAN) for conditional image synthesis and we name the proposed model as ArtGAN. One of the key innovation of ArtGAN is that, the gradient of the loss function w.r.t. the label (randomly assigned to each generated image) is back-propagated from the categorical discriminator to the generator. With the feedback from the label information, the generator is able to learn more efficiently and generate image with better quality. Inspired by recent works, an autoencoder is incorporated into the categorical discriminator for additional complementary information. Last but not least, we introduce a novel strategy to improve the image quality. In the experiments, we evaluate ArtGAN on CIFAR-10 and STL-10 via ablation studies. The empirical results showed that our proposed model outperforms the state-of-the-art results on CIFAR-10 in terms of Inception score. Qualitatively, we demonstrate that ArtGAN is able to generate plausible-looking images on Oxford-102 and CUB-200, as well as able to draw realistic artworks based on style, artist, and genre. The source code and models are available at: https://github.com/cs-chan/ArtGAN

Motivation & Objective

• To improve conditional image synthesis in GANs by leveraging label information more effectively during training.
• To generate high-fidelity natural images and complex artworks, including abstract and non-figurative styles, from conditional inputs.
• To address the challenge of learning artistic representations that are not easily captured by standard GANs due to their abstract or stylistic nature.
• To develop a computationally efficient method for enhancing image quality without training multiple GANs.
• To demonstrate that the model generalizes beyond memorization by generating plausible, smooth interpolations in latent space.

Proposed method

• The generator takes a noise vector and a randomly assigned label as input, with the label used to compute the loss function.
• Label gradients are backpropagated from the categorical discriminator to the generator, enabling more efficient learning of class-specific features.
• A shared encoder is used in a categorical autoencoder-based discriminator, which jointly performs class prediction, adversarial training, and image reconstruction.
• A novel image quality enhancement strategy upsamples generated images to double resolution, then applies average pooling to downsample, simulating a voting mechanism for improved pixel quality.
• The architecture, named ArtGAN-AEM, integrates the autoencoder and discriminator with shared weights to reduce computational cost.
• Latent space interpolation is performed to evaluate generalization and avoid memorization of training data.

Experimental results

Research questions

• RQ1 Can backpropagating label gradients improve the generator’s ability to learn class-specific visual features in conditional GANs?
• RQ2 Can integrating an autoencoder into the discriminator enhance feature representation and image quality in conditional image synthesis?
• RQ3 Does the proposed resolution-upscaling and average-pooling voting strategy effectively improve generated image quality without training multiple GANs?
• RQ4 Can the model generate realistic and diverse artworks across different styles, genres, and artists, including abstract and non-figurative forms?
• RQ5 Does the model generalize beyond memorization, as evidenced by smooth latent space interpolations?

Key findings

• ArtGAN-AEM achieves a state-of-the-art Inception score on CIFAR-10, outperforming previous methods.
• The model generates high-quality, plausible images on the Oxford-102 and CUB-200 datasets, with clear object structures and realistic details.
• ArtGAN successfully learns and reproduces distinct artistic styles, such as the dark tones of Baroque, the lightness of Rococo, and the yellowish hue of Ukiyo-e prints.
• Synthesized paintings of artists like Vincent van Gogh, Eugene Boudin, and Ivan Shishkin reflect characteristic styles, such as expressive brushwork, marine themes, and forest landscapes, respectively.
• Latent space interpolations show smooth, plausible transitions between generated images, confirming that the model does not memorize training data.
• The model generates realistic artworks based on style, genre, and artist, demonstrating its ability to learn complex, non-figurative representations from the Wikiart dataset.