[Paper Review] EFANet: Exchangeable Feature Alignment Network for Arbitrary Style Transfer

Zhijie Wu, Chunjin Song+3 more|arXiv (Cornell University)|Nov 26, 2018

Generative Adversarial Networks and Image Synthesis44 references4 citations

TL;DR

EFANet proposes a novel end-to-end framework for arbitrary style transfer that jointly aligns content and style features by learning exchangeable features through a Feature Exchange Block, significantly improving stylization quality and structure preservation. By introducing a whitening loss to purify content features and enabling multi-scale fusion, EFANet achieves real-time, high-fidelity stylization superior to prior methods in both qualitative and quantitative benchmarks.

ABSTRACT

Style transfer has been an important topic both in computer vision and graphics. Since the seminal work of Gatys et al. first demonstrates the power of stylization through optimization in the deep feature space, quite a few approaches have achieved real-time arbitrary style transfer with straightforward statistic matching techniques. In this work, our key observation is that only considering features in the input style image for the global deep feature statistic matching or local patch swap may not always ensure a satisfactory style transfer; see e.g., Figure 1. Instead, we propose a novel transfer framework, EFANet, that aims to jointly analyze and better align exchangeable features extracted from content and style image pair. In this way, the style features from the style image seek for the best compatibility with the content information in the content image, leading to more structured stylization results. In addition, a new whitening loss is developed for purifying the computed content features and better fusion with styles in feature space. Qualitative and quantitative experiments demonstrate the advantages of our approach.

Motivation & Objective

• To address the limitation of existing style transfer methods that rely on fixed or non-adaptive style statistics, leading to distorted or inconsistent stylization under complex styles.
• To improve stylization quality by jointly analyzing content and style images to extract pair-specific, compatible style features.
• To develop a method that enables arbitrary style transfer in real-time while preserving content structure and mimicking complex style patterns accurately.
• To enhance feature fusion by removing style artifacts from content features through a novel whitening loss function.

Proposed method

• A Feature Exchange Block is introduced to learn common, exchangeable style features between content and style images by maximizing their shared representation in deep feature space.
• A novel whitening loss is applied to content features to remove style patterns, enabling cleaner fusion with style features and improving compatibility.
• Multi-scale feature fusion is employed across multiple layers (e.g., relu_4) to better capture both global and local style patterns.
• The network is trained end-to-end with a combination of content loss, style loss, and the proposed whitening loss to optimize stylization quality.
• The framework supports real-time inference by replacing iterative optimization with a feed-forward network, enabling fast stylization on a single GPU.
• A blending strategy using a hyperparameter α allows control over stylization intensity, enabling smooth transitions from content to stylized image.

Experimental results

Research questions

• RQ1 Can joint analysis of content and style images improve the quality and consistency of arbitrary style transfer compared to methods using isolated style statistics?
• RQ2 Does learning exchangeable features that are common to both content and style images lead to better alignment and more structured stylization results?
• RQ3 Can a whitening loss effectively purify content features by removing residual style patterns, thereby improving fusion with target style features?
• RQ4 How does multi-scale feature fusion impact the preservation of fine-grained style patterns and spatial structure in stylized outputs?
• RQ5 To what extent does the proposed framework outperform existing real-time style transfer methods in visual quality and user preference?

Key findings

• EFANet achieves the highest user preference score (600 votes from 30 subjects) in a pairwise comparison, outperforming all baselines in perceptual quality.
• The full model with common feature learning reduces style loss by a significant margin compared to the ablated baseline without exchangeable features, demonstrating improved style matching.
• The ablation study confirms that removing the whitening loss leads to blurred contours and yellowish artifacts, proving its role in enhancing feature compatibility and visual clarity.
• The multi-scale strategy significantly improves the capture of salient style patterns, as shown in visual comparisons where the full model preserves texture and color consistency better than single-scale alternatives.
• EFANet achieves real-time inference with a 12GB Titan V GPU, maintaining performance comparable to AdaIN while delivering superior stylization quality.
• The model supports flexible stylization control via blending (α), enabling smooth transitions from content to stylized image without retraining.