[Paper Review] Texture Networks: Feed-forward Synthesis of Textures and Stylized Images

Dmitry Ulyanov, Vadim Lebedev+2 more|arXiv (Cornell University)|Mar 10, 2016

Generative Adversarial Networks and Image Synthesis16 references605 citations

TL;DR

Texture Networks train compact feed-forward generators to synthesize textures and apply style transfer from a single example, yielding similar quality to optimization-based methods but with much faster, memory-efficient results.

ABSTRACT

Gatys et al. recently demonstrated that deep networks can generate beautiful textures and stylized images from a single texture example. However, their methods requires a slow and memory-consuming optimization process. We propose here an alternative approach that moves the computational burden to a learning stage. Given a single example of a texture, our approach trains compact feed-forward convolutional networks to generate multiple samples of the same texture of arbitrary size and to transfer artistic style from a given image to any other image. The resulting networks are remarkably light-weight and can generate textures of quality comparable to Gatys~et~al., but hundreds of times faster. More generally, our approach highlights the power and flexibility of generative feed-forward models trained with complex and expressive loss functions.

Motivation & Objective

• Motivate a fast alternative to slow optimization-based texture synthesis and style transfer methods.
• Propose a lightweight, fully convolutional, multi-scale generator that maps noise to textures.
• Demonstrate training of generators using statistics from a fixed pre-trained descriptor network (Gram matrices) as loss signals.
• Show that the approach yields comparable texture quality to prior methods while offering substantial speedups and memory benefits.
• Extend the generator to perform style transfer by combining texture loss with content loss.

Proposed method

• Train a feed-forward generator g that maps noise z to an image x by minimizing a texture loss L_T based on Gram matrix statistics from a fixed descriptor CNN (e.g., VGG).
• Use a multi-scale, fully-convolutional architecture with upsampling and scale-wise noise inputs to synthesize textures of arbitrary size.
• Train the generator end-to-end with SGD, evaluating the descriptor to compute G^l(x) and the loss against the reference texture x_0.
• Extend the generator for style transfer by feeding both content y and noise z, and training with a weighted sum of texture loss L_T and content loss L_C.
• For stylization, concatenate multi-scale noise with downsampled content and increase the number of scales to improve results.
• Train using Adam with a small image pool, and demonstrate real-time capable synthesis (approximately 20 ms per 256x256 image).

Experimental results

Research questions

• RQ1 Can a compact, feed-forward network learn to synthesize textures of comparable quality to optimization-based methods?
• RQ2 Can the same framework be extended to real-time style transfer by combining texture statistics with content constraints?
• RQ3 What architectural choices (multi-scale, concatenation, normalization) best enable high-quality texture generation with few parameters?
• RQ4 How does the speed and memory usage of a trained generator compare to iterative optimization methods for texture synthesis and stylization?

Key findings

• A single feed-forward generator can synthesize textures with quality and diversity comparable to optimization-based methods like Gatys et al., but hundreds of times faster.
• The proposed generator achieves approximately 500x speed-ups over iterative optimization and uses far less memory (about 170 MB vs 1100 MB for a 256x256 sample).
• A compact multi-scale architecture with ~65K parameters yields textures at arbitrary sizes, trained end-to-end using Gram-matrix based texture loss.
• For style transfer, combining texture loss with a content loss yields visually compelling stylizations comparable to optimization-based methods on many styles, though some cases are less impressive.
• Fully convolutional design allows stylization of larger images beyond training resolution (e.g., 256x256 trained networks applied to 1024x768 results).
• Training runs efficiently (about two hours for a model on a K40), and test-time stylizations run in around 20 ms per image.