[Paper Review] Fast Universal Style Transfer for Artistic and Photorealistic Rendering

Jie An, Haoyi Xiong+3 more|arXiv (Cornell University)|Jul 6, 2019

Generative Adversarial Networks and Image Synthesis42 references4 citations

TL;DR

This paper proposes ArtNet and PhotoNet, two novel neural network architectures built on multi-layer autoencoders for fast, single-pass universal style transfer in both artistic and photorealistic rendering. By integrating feature aggregation and normalized skip connections with pre-defined style transfer modules (e.g., WCT, AdaIN), the methods achieve state-of-the-art stylization quality with 3× to 100× speedup over prior methods, significantly reducing artifacts, distortions, and computational cost while preserving image details.

ABSTRACT

Universal style transfer is an image editing task that renders an input content image using the visual style of arbitrary reference images, including both artistic and photorealistic stylization. Given a pair of images as the source of content and the reference of style, existing solutions usually first train an auto-encoder (AE) to reconstruct the image using deep features and then embeds pre-defined style transfer modules into the AE reconstruction procedure to transfer the style of the reconstructed image through modifying the deep features. While existing methods typically need multiple rounds of time-consuming AE reconstruction for better stylization, our work intends to design novel neural network architectures on top of AE for fast style transfer with fewer artifacts and distortions all in one pass of end-to-end inference. To this end, we propose two network architectures named ArtNet and PhotoNet to improve artistic and photo-realistic stylization, respectively. Extensive experiments demonstrate that ArtNet generates images with fewer artifacts and distortions against the state-of-the-art artistic transfer algorithms, while PhotoNet improves the photorealistic stylization results by creating sharp images faithfully preserving rich details of the input content. Moreover, ArtNet and PhotoNet can achieve 3X to 100X speed-up over the state-of-the-art algorithms, which is a major advantage for large content images.

Motivation & Objective

• To address the high computational cost and quality degradation (artifacts, distortions) in existing universal style transfer methods that rely on multi-round reconstruction and post-processing.
• To enable fast, end-to-end style transfer with a single inference pass, eliminating the need for iterative refinement or time-consuming optimization.
• To improve stylization quality—especially in detail preservation and sharpness—for both artistic and photorealistic rendering using enhanced autoencoder architectures.
• To achieve superior performance over state-of-the-art methods like WCT, AdaIN, and PhotoWCT in both visual quality and inference speed.

Proposed method

• Proposes ArtNet, a feed-forward network that aggregates multi-level features from the encoder and applies a multi-stage style transfer module (MST) to enhance artistic stylization with reduced artifacts.
• Introduces PhotoNet, which uses normalized skip connections to preserve fine details during photorealistic stylization, improving sharpness and fidelity to the original content.
• Employs a pre-trained autoencoder with improved reconstruction capacity to minimize image distortion and artifacts before style transfer.
• Applies pre-defined style transfer modules (e.g., WCT, AdaIN) directly in the decoder or skip connections, avoiding iterative reconstruction loops.
• Uses feature aggregation across low- and high-level features to better preserve structural and texture details during stylization.
• Employs a single forward pass for inference, drastically reducing computation time compared to multi-round methods.

Experimental results

Research questions

• RQ1 Can a single-pass, end-to-end neural network architecture outperform multi-round style transfer methods in both speed and visual quality?
• RQ2 Can feature aggregation and normalized skip connections reduce artifacts and distortions in artistic and photorealistic stylization, respectively?
• RQ3 To what extent can improved autoencoder reconstruction capacity enhance stylization performance without iterative refinement?
• RQ4 How does the integration of style transfer modules in early vs. late decoder stages affect image quality and detail preservation?

Key findings

• ArtNet reduces artifacts and distortions compared to WCT and AdaIN, producing visually superior artistic stylization results in a single inference pass.
• PhotoNet generates sharper, more detailed images than PhotoWCT, with higher total variation scores indicating better preservation of fine details.
• The proposed methods achieve 3× to 100× speedup over state-of-the-art algorithms, especially benefiting large-content images.
• Image reconstruction error is significantly lower for ArtNet and PhotoNet (74,643.66 and 75,021.59) than for WCT (83,196.87) and AdaIN (86,611.57), indicating better feature fidelity.
• FID scores show PhotoNet outperforms PhotoWCT, indicating better distributional similarity to reference style images in photorealistic transfer.
• Ablation studies confirm that feature aggregation (FA) and normalized skip connections (NS) are critical components, with MST-∞ (all stages) yielding the best visual results.