[Paper Review] Fast Patch-based Style Transfer of Arbitrary Style

Tian Qi Chen, Mark Schmidt|arXiv (Cornell University)|Dec 13, 2016

Generative Adversarial Networks and Image Synthesis18 references222 citations

TL;DR

Proposes a fast, arbitrary-style transfer method using a single-layer CNN style-swap objective and an inverse network for efficient feedforward stylization across arbitrary content and style images, including video.

ABSTRACT

Artistic style transfer is an image synthesis problem where the content of an image is reproduced with the style of another. Recent works show that a visually appealing style transfer can be achieved by using the hidden activations of a pretrained convolutional neural network. However, existing methods either apply (i) an optimization procedure that works for any style image but is very expensive, or (ii) an efficient feedforward network that only allows a limited number of trained styles. In this work we propose a simpler optimization objective based on local matching that combines the content structure and style textures in a single layer of the pretrained network. We show that our objective has desirable properties such as a simpler optimization landscape, intuitive parameter tuning, and consistent frame-by-frame performance on video. Furthermore, we use 80,000 natural images and 80,000 paintings to train an inverse network that approximates the result of the optimization. This results in a procedure for artistic style transfer that is efficient but also allows arbitrary content and style images.

Motivation & Objective

• Motivate efficient artistic style transfer that works with arbitrary content and style images, avoiding per-style retraining.
• Introduce a single-layer objective that combines content and style statistics for easier optimization.
• Develop a style-swap operation that swaps content activations with style patches and can be implemented as standard convolutions.
• Train an inverse network to invert activations and enable fast, feedforward style transfer for new style images.

Proposed method

• Define a patch-based style swap in a single CNN layer that replaces content patches with closest-matching style patches using normalized cross-correlation.
• Implement style swap as a network with three operations: 2D convolution, channel-wise argmax, and 2D transposed convolution.
• Optimize for stylized activations by minimizing the Frobenius norm between Phi(I) and Phiss(C,S) plus total-variation regularization.
• Train an inverse network to approximate the optimal inverse of a truncated VGG-19, with augmented training including style-swapped activations.
• Use the inverse network in a feedforward pipeline: compute Phi(C) and Phi(S), obtain Phi^{ss}(C,S) via style swap, then generate the image with the inverse network.
• Demonstrate that the approach supports arbitrary styles without retraining and enables consistent frame-by-frame results for video.

Experimental results

Research questions

• RQ1 Can a single-layer CNN activation objective produce competitive style transfer results with arbitrary content and style images?
• RQ2 Does a patch-based style swap yield visually pleasing results and stable video stylization without explicit temporal constraints?
• RQ3 Can an inverse network generalize to stylizations with new style images and varied content sizes?
• RQ4 How does the proposed method compare to optimization- and other feedforward-based style transfer in speed and flexibility?

Key findings

• The method achieves visually pleasing results by swapping content activations with style patches in a single CNN layer (relu3_1) and provides intuitive tuning via patch size.
• The style swap implementation can be realized with standard convolutional and transposed convolution layers, enabling efficient computation.
• The inverse network can approximate the optimization optimum and generalize to new content and style images, enabling a fast feedforward stylization pipeline.
• Compared times show that optimization-based Gatys et al. style transfer is slower than the proposed style swap approach, and the inverse network offers faster results than optimization in some settings.
• The approach allows arbitrary style images without retraining a fixed-style network, and supports video with frame-consistent stylization.