[Paper Review] Photorealistic Style Transfer via Wavelet Transforms

Jaejun Yoo, Youngjung Uh+3 more|arXiv (Cornell University)|Mar 23, 2019

Generative Adversarial Networks and Image Synthesis35 references37 citations

TL;DR

The paper introduces WCT 2, a wavelet-corrected, end-to-end photorealistic style transfer model that preserves content structure, enables progressive stylization in a single pass, and runs at high resolution with no post-processing.

ABSTRACT

Recent style transfer models have provided promising artistic results. However, given a photograph as a reference style, existing methods are limited by spatial distortions or unrealistic artifacts, which should not happen in real photographs. We introduce a theoretically sound correction to the network architecture that remarkably enhances photorealism and faithfully transfers the style. The key ingredient of our method is wavelet transforms that naturally fits in deep networks. We propose a wavelet corrected transfer based on whitening and coloring transforms (WCT$^2$) that allows features to preserve their structural information and statistical properties of VGG feature space during stylization. This is the first and the only end-to-end model that can stylize a $1024 imes1024$ resolution image in 4.7 seconds, giving a pleasing and photorealistic quality without any post-processing. Last but not least, our model provides a stable video stylization without temporal constraints. Our code, generated images, and pre-trained models are all available at https://github.com/ClovaAI/WCT2.

Motivation & Objective

• Motivate photorealistic style transfer that preserves fine content details while applying a reference style.
• Replace lossy pooling/unpooling in VGG-based encoders/decoders with Haar wavelet pooling to enable exact reconstruction.
• Propose progressive stylization within a single forward pass to improve efficiency and reduce artifacts.
• Demonstrate high-resolution (1024×1024) stylization with minimal runtime and no post-processing steps.
• Show temporal stability for video stylization without temporal regularization.

Proposed method

• Replace max-pooling/unpooling in VGG encoder/decoder with Haar wavelet pooling/unpooling to enable exact signal reconstruction.
• Use whitening and coloring transforms (WCT) for style transfer in the VGG feature space with a progressive single-pass strategy.
• Stylize progressively across encoder levels (conv1_X to conv4_X) within one forward pass, avoiding multi-level decoders.
• Optionally allow augmentation with multi-level stylization to increase style strength while monitoring artifacts.
• Train a single encoder-decoder pair; leverage semantic maps to guide region-wise style transfer; implement using an end-to-end pipeline (WCT 2).
• Grounding in frame theory to justify perfect reconstruction with tight wavelet frames and energy compaction for stable stylization.

Experimental results

Research questions

• RQ1 Can wavelet-based pooling/unpooling enable exact reconstruction to preserve content structure during photorealistic style transfer?
• RQ2 Does progressive, single-pass stylization yield equal or better photorealism and stability compared to multi-level approaches?
• RQ3 What are the quantitative and qualitative gains in high-resolution stylization without post-processing?
• RQ4 How does WCT 2 perform in video stylization regarding temporal stability without temporal constraints?

Key findings

• WCT 2 achieves photorealistic stylization with no post-processing unlike prior methods.
• At 1024×1024, WCT 2 runs in 4.7 seconds, considerably faster than DPST and PhotoWCT (full), with lower memory usage.
• WCT 2 yields higher SSIM and lower Gram-style losses than baselines, indicating better photorealism and style fidelity.
• Progressive stylization within a single pass reduces error amplification and enables a single encoder-decoder, improving efficiency.
• Wavelet pooling preserves content details and edges, enabling selective stylization when transforming LL vs high-frequency components.
• Video stylization with WCT 2 demonstrates temporal stability without optical-flow constraints.