[Paper Review] ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows

Jie An, Siyu Huang+4 more|arXiv (Cornell University)|Mar 31, 2021

Generative Adversarial Networks and Image Synthesis61 references21 citations

TL;DR

ArtFlow proposes a reversible neural flow-based framework to eliminate content leak in universal style transfer by enabling lossless, unbiased forward and backward inference. It replaces autoencoder-based feature extraction with a projection-transfer-reversion pipeline using a learnable, invertible network, achieving state-of-the-art performance without degradation over multiple stylization rounds.

ABSTRACT

Universal style transfer retains styles from reference images in content images. While existing methods have achieved state-of-the-art style transfer performance, they are not aware of the content leak phenomenon that the image content may corrupt after several rounds of stylization process. In this paper, we propose ArtFlow to prevent content leak during universal style transfer. ArtFlow consists of reversible neural flows and an unbiased feature transfer module. It supports both forward and backward inferences and operates in a projection-transfer-reversion scheme. The forward inference projects input images into deep features, while the backward inference remaps deep features back to input images in a lossless and unbiased way. Extensive experiments demonstrate that ArtFlow achieves comparable performance to state-of-the-art style transfer methods while avoiding content leak.

Motivation & Objective

• Address the content leak phenomenon in state-of-the-art universal style transfer methods, where repeated stylization degrades content fidelity.
• Identify the root causes of content leak in existing methods, including reconstruction errors and biased training in decoders and style transfer modules.
• Develop a reversible, unbiased feature extraction and reconstruction mechanism to enable fair comparison and robust stylization across multiple rounds.
• Enable lossless content-style separation and reversion by leveraging invertible neural flows for both forward and backward inference.
• Achieve comparable visual quality to SOTA methods while eliminating content corruption over iterative stylization.

Proposed method

• Introduce a Projection Flow Network (PFN) based on neural flows, inspired by GLOW, using invertible $1\times1$ convolutions, activation normalization, and affine coupling layers.
• Implement a projection-transfer-reversion pipeline: forward inference projects content and style images into deep features, followed by unbiased style transfer, then backward inference reconstructs images losslessly.
• Use the PFN to enable both forward and reverse inferences, ensuring that feature extraction and image recovery are free from reconstruction error and bias.
• Apply the PFN to replace the VGG19-based autoencoder in existing methods (e.g., AdaIN, WCT, Avatar-Net), enabling unbiased stylization without architectural changes to the style transfer module.
• Perform theoretical and empirical analysis of style transfer module biases, identifying that Avatar-Net’s style decorator is biased toward style, while AdaIN and WCT are unbiased.
• Integrate the unbiased modules (AdaIN, WCT) with PFN to form ArtFlow, ensuring fair and stable performance across multiple stylization iterations.

Figure 1: Content leak visualization. Existing style transfer methods are not effective to preserve image content after several rounds of stylization process as shown in (d), although their performance is state-of-the-art in the first round as shown in (c).

Experimental results

Research questions

• RQ1 What causes content leak in existing universal style transfer methods during repeated stylization?
• RQ2 Can a reversible, invertible neural network eliminate reconstruction errors and bias in feature extraction and image recovery?
• RQ3 How does the proposed PFN-based framework compare to autoencoder-based methods in preserving content fidelity over multiple stylization rounds?
• RQ4 Which style transfer modules (AdaIN, WCT, Avatar-Net) are inherently unbiased, and how does this affect performance in the ArtFlow framework?
• RQ5 Can ArtFlow achieve comparable visual quality to SOTA methods while eliminating content leak?

Key findings

• ArtFlow successfully prevents content leak in universal style transfer, even after 20 rounds of stylization, unlike vanilla AdaIN, WCT, and Avatar-Net, which show severe content degradation.
• The PFN-based framework enables lossless and unbiased image reconstruction, as demonstrated by the sharpness and fidelity of inverted content features compared to AdaIN and WCT.
• User studies show ArtFlow received 314 votes out of 799, ranking highest among all methods, indicating its visual quality is comparable or superior to SOTA approaches.
• Theoretical and empirical analysis confirms that AdaIN and WCT modules are unbiased, while Avatar-Net’s style decorator is biased toward style, explaining its persistent content leak even with PFN.
• ArtFlow+WCT is faster than vanilla WCT due to the elimination of multi-level stylization, and ArtFlow+AdaIN avoids content leak while maintaining high performance.
• Content-style separation via ArtFlow produces cleaner, sharper content representations with minimal style artifacts, unlike AdaIN and WCT, which retain residual style effects.

Figure 2: A comparison between the auto-encoder based style transfer framework and the proposed ArtFlow framework.