[Paper Review] Texture Synthesis Using Convolutional Neural Networks

Leon A. Gatys, Alexander S. Ecker+1 more|arXiv (Cornell University)|May 27, 2015

Generative Adversarial Networks and Image Synthesis28 references425 citations

TL;DR

This paper proposes a novel parametric texture synthesis model using feature representations from a pre-trained convolutional neural network (VGG-19), where textures are generated by optimizing a noise image to match the Gram matrices of feature maps across multiple layers. The method produces high-quality, perceptually realistic textures that surpass previous parametric models, while also revealing how hierarchical features in deep networks encode increasingly explicit object information.

ABSTRACT

Here we introduce a new model of natural textures based on the feature spaces of convolutional neural networks optimised for object recognition. Samples from the model are of high perceptual quality demonstrating the generative power of neural networks trained in a purely discriminative fashion. Within the model, textures are represented by the correlations between feature maps in several layers of the network. We show that across layers the texture representations increasingly capture the statistical properties of natural images while making object information more and more explicit. The model provides a new tool to generate stimuli for neuroscience and might offer insights into the deep representations learned by convolutional neural networks.

Motivation & Objective

• To develop a new parametric texture model that surpasses existing methods in perceptual quality.
• To leverage the hierarchical feature spaces of a pre-trained CNN for texture representation and synthesis.
• To explore how deep features in CNNs encode both texture statistics and object identity information.
• To provide a new tool for neuroscience by generating stimuli based on deep network representations.

Proposed method

• The model uses Gram matrices of feature maps from multiple convolutional layers of a pre-trained VGG-19 network to represent texture statistics.
• Texture synthesis is performed by optimizing a noise image through gradient descent to match the Gram matrices of the original texture across selected layers.
• The loss function is a weighted sum of layer-specific reconstruction errors based on Gram matrix differences.
• Only convolutional layers are used, preserving spatial invariance and enabling synthesis of arbitrarily sized textures.
• The method relies on the hierarchical, non-linear feature extraction of CNNs, with feature maps from deeper layers capturing more complex statistical patterns.
• The optimization process is iterative and differentiable, allowing for high-fidelity texture generation.

Experimental results

Research questions

• RQ1 Can deep convolutional neural network features be used to define a parametric texture model that produces perceptually realistic textures?
• RQ2 How do the statistical representations of texture in CNN feature spaces compare to traditional models like Portilla-Simoncelli?
• RQ3 To what extent do higher layers in a CNN preserve object identity while encoding texture statistics?
• RQ4 Can the Gram matrix of CNN features serve as a stationary, invariant representation for texture synthesis and analysis?

Key findings

• The proposed method generates textures of perceptual quality comparable to non-parametric methods, significantly outperforming previous parametric models.
• Synthesized textures are indistinguishable from real textures in human perception tests, demonstrating the model's generative power.
• The Gram matrix representation across layers captures increasingly complex statistical properties of natural images, with deeper layers encoding more explicit object identity.
• Linear decoding from the final pooling layer's Gram matrix achieves 87.7% top-5 accuracy, close to the original network's 88.6% performance.
• The model reveals that object identity information is preserved in the feature space independently of spatial layout, consistent with shift-equivariance in CNNs.
• The method is computationally intensive but benefits from ongoing advances in deep learning inference, suggesting future scalability.