[Paper Review] AesUST: Towards Aesthetic-Enhanced Universal Style Transfer

Zhizhong Wang, Zhanjie Zhang+5 more|arXiv (Cornell University)|Aug 27, 2022

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

AesUST proposes a novel aesthetic-enhanced universal style transfer framework that integrates learned human-delightful aesthetic features into style transfer via a dedicated aesthetic discriminator and an Aesthetic-aware Style-Attention (AesSA) module, significantly improving visual harmony and realism. Experiments and user studies show AesUST generates results indistinguishable from real artist-created paintings, outperforming SOTA methods in both perceptual quality and aesthetic realism.

ABSTRACT

Recent studies have shown remarkable success in universal style transfer which transfers arbitrary visual styles to content images. However, existing approaches suffer from the aesthetic-unrealistic problem that introduces disharmonious patterns and evident artifacts, making the results easy to spot from real paintings. To address this limitation, we propose AesUST, a novel Aesthetic-enhanced Universal Style Transfer approach that can generate aesthetically more realistic and pleasing results for arbitrary styles. Specifically, our approach introduces an aesthetic discriminator to learn the universal human-delightful aesthetic features from a large corpus of artist-created paintings. Then, the aesthetic features are incorporated to enhance the style transfer process via a novel Aesthetic-aware Style-Attention (AesSA) module. Such an AesSA module enables our AesUST to efficiently and flexibly integrate the style patterns according to the global aesthetic channel distribution of the style image and the local semantic spatial distribution of the content image. Moreover, we also develop a new two-stage transfer training strategy with two aesthetic regularizations to train our model more effectively, further improving stylization performance. Extensive experiments and user studies demonstrate that our approach synthesizes aesthetically more harmonious and realistic results than state of the art, greatly narrowing the disparity with real artist-created paintings. Our code is available at https://github.com/EndyWon/AesUST.

Motivation & Objective

• To address the aesthetic-unrealistic problem in universal style transfer, where generated images suffer from disharmonious patterns and artifacts.
• To improve the realism and visual harmony of stylized images by incorporating universal human-delightful aesthetic features from artist-created paintings.
• To develop a method that balances content preservation and style transfer while ensuring aesthetic consistency.
• To train the model effectively using a two-stage strategy with dual aesthetic regularizations.
• To narrow the perceptual gap between AI-generated stylized images and real artist-created artworks.

Proposed method

• Introduces an aesthetic discriminator trained on a large corpus of artist-created paintings to extract universal human-delightful aesthetic features.
• Proposes the Aesthetic-aware Style-Attention (AesSA) module to dynamically integrate style patterns based on the global aesthetic distribution of the style image and local semantic spatial distribution of the content image.
• Employs a two-stage training strategy: stage I pre-trains the generator with content and style reconstruction, stage II fine-tunes with adversarial and aesthetic regularization.
• Applies two aesthetic regularizations: one to preserve content structure and style transfer quality, another to enhance perceptual style pattern fidelity.
• Uses adversarial loss to encourage realistic, artifact-free outputs by distinguishing real paintings from generated ones.
• Leverages the aesthetic discriminator not only as a discriminator but also as a feature extractor for guiding style integration.

Figure 2. Overview of our proposed AesUST. Note that we apply a two-stage transfer training strategy with a transfer learning fashion: (1) At stage I, the aesthetic discriminator only acts as the discriminator, and we pre-train the generator (AesSA module and decoder) using VGG features only. (2) At

Experimental results

Research questions

• RQ1 Can a learned aesthetic discriminator effectively capture universal human-delightful features from artist-created paintings to guide style transfer?
• RQ2 How can aesthetic features be integrated into the style transfer process to improve visual harmony and realism?
• RQ3 Does the proposed two-stage training strategy with aesthetic regularizations lead to better stylization performance than end-to-end training?
• RQ4 To what extent does AesUST reduce the perceptual gap between AI-generated stylized images and real artist-created paintings?
• RQ5 Can the AesSA module flexibly balance global aesthetic consistency and local semantic fidelity in style transfer?

Key findings

• AesUST achieves the highest deception score (87.4%) in user studies, closest to real paintings (89.2%), indicating near-indistinguishability from real artworks.
• In A/B preference tests, AesUST received 68.7% of votes, significantly outperforming SOTA methods like AdaIN (52.3%) and LST (55.1%) in user preference.
• The ablation study confirms that removing the adversarial loss leads to severe artifacts and disharmonious patterns, validating its role in realism.
• Removing the first aesthetic regularization (L_AR1) degrades both content preservation and style transfer, while removing the second (L_AR2) reduces style fidelity, proving their necessity.
• The two-stage training strategy is essential: stage I pre-training enables effective style integration, and stage II fine-tuning with aesthetic guidance improves final quality.
• AesUST runs in real time at ~25 FPS on 512×512 images, matching the efficiency of SOTA methods like AdaIN and LST.

Figure 3. Aesthetic-aware style-attention (AesSA) module. $F_{c}$ , $F_{s}$ , and $F_{a}$ are content, style, and aesthetic features, respectively. “ $Norm$ ” denotes the mean-variance channel-wise normalization.