[Paper Review] Texture Synthesis with Spatial Generative Adversarial Networks

Nikolay Jetchev, Urs Bergmann+1 more|arXiv (Cornell University)|Nov 24, 2016

Generative Adversarial Networks and Image Synthesis14 references108 citations

TL;DR

The paper introduces Spatial GAN (SGAN), a fully convolutional GAN that maps a spatial noise tensor to large texture images, enabling high-quality, scalable, and real-time texture synthesis without fully connected layers.

ABSTRACT

Generative adversarial networks (GANs) are a recent approach to train generative models of data, which have been shown to work particularly well on image data. In the current paper we introduce a new model for texture synthesis based on GAN learning. By extending the input noise distribution space from a single vector to a whole spatial tensor, we create an architecture with properties well suited to the task of texture synthesis, which we call spatial GAN (SGAN). To our knowledge, this is the first successful completely data-driven texture synthesis method based on GANs. Our method has the following features which make it a state of the art algorithm for texture synthesis: high image quality of the generated textures, very high scalability w.r.t. the output texture size, fast real-time forward generation, the ability to fuse multiple diverse source images in complex textures. To illustrate these capabilities we present multiple experiments with different classes of texture images and use cases. We also discuss some limitations of our method with respect to the types of texture images it can synthesize, and compare it to other neural techniques for texture generation.

Motivation & Objective

• Motivate and address limitations of existing texture synthesis methods for large-scale, high-quality textures.
• Propose a fully convolutional GAN architecture that accepts a spatial noise tensor to generate textures of arbitrary size.
• Demonstrate SGAN’s ability to fuse multiple source images and produce seamless textures.
• Evaluate generation speed and quality across various texture types and sizes.
• Discuss limitations and potential extensions of SGAN for broader texture-related tasks.

Proposed method

• Extend DCGAN into a spatially structured generator G that maps a tensor Z in R^{l x m x d} to an image X in R^{h x w x 3}.
• Discriminator D outputs a 2D field of fake/real probabilities across spatial locations, enabling training over all patches simultaneously.
• Train G and D with a joint objective V(D,G) averaged over all spatial locations, using standard GAN loss with the G=D(G(Z)) trick.
• Enforce a fully convolutional architecture with no fully connected layers to enable arbitrary output sizes.
• Utilize patches X′ sampled from training textures I as real data, ensuring h=w=patch size when needed for training stability.
• Analyze projective fields and receptive fields to relate input Z to output pixels and discuss implications for texture properties.

Experimental results

Research questions

• RQ1 Can a fully convolutional GAN learn and synthesize textures of arbitrary size by conditioning on a spatial noise tensor?
• RQ2 How does SGAN perform in terms of realism, scalability, and speed compared to prior texture synthesis methods like Gatys et al. and non-GAN approaches?
• RQ3 What capabilities arise from combining multiple source textures, and can SGAN produce seamless textures or composite textures?
• RQ4 What are the limitations of SGAN regarding regular patterns or long-range dependencies beyond its projective field?
• RQ5 How do network depth and receptive field affect the quality and structure of learned textures?

Key findings

• SGAN enables real-time generation of high-quality textures with a single forward pass.
• SGAN can output textures of any desired size due to its fully convolutional design.
• Increasing SGAN depth (e.g., SGAN4, SGAN5, SGAN6) enlarges the projective field and improves structure such as longer street-like patterns in satellite textures.
• SGAN can fuse multiple source images into composite textures, illustrating natural blending of diverse textures.
• SGAN generation speeds scale sublinearly with image size and outperform TextureNet and Gatys et al. in reported benchmarks at common resolutions.
• For satellite textures (Barcelona), SGAN better preserves directional structure than Gatys et al., as evidenced by spatial autocorrelation analyses.