[Paper Review] GANosaic: Mosaic Creation with Generative Texture Manifolds

Nikolay Jetchev, Urs Bergmann+1 more|arXiv (Cornell University)|Dec 1, 2017

Generative Adversarial Networks and Image Synthesis10 references7 citations

TL;DR

GANosaic introduces a novel method for generating high-resolution, seamless mosaics by optimizing the latent noise space of a Periodic Spatial GAN (PSGAN) to synthesize textures that match a content image. By combining a perceptual content loss with a differentiable statistical regularization over local noise channels, it produces aesthetically pleasing, morphing mosaics without visible tile boundaries, enabling large-scale, memory-efficient image generation with linear runtime scaling.

ABSTRACT

This paper presents a novel framework for generating texture mosaics with convolutional neural networks. Our method is called GANosaic and performs optimization in the latent noise space of a generative texture model, which allows the transformation of a content image into a mosaic exhibiting the visual properties of the underlying texture manifold. To represent that manifold, we use a state-of-the-art generative adversarial method for texture synthesis, which can learn expressive texture representations from data and produce mosaic images with very high resolution. This fully convolutional model generates smooth (without any visible borders) mosaic images which morph and blend different textures locally. In addition, we develop a new type of differentiable statistical regularization appropriate for optimization over the prior noise space of the PSGAN model.

Motivation & Objective

• To create seamless, high-resolution mosaics that blend multiple textures naturally, avoiding visible tile boundaries.
• To enable the generation of novel, morphing textures from a manifold learned across multiple example images.
• To develop a differentiable regularization method that keeps optimized noise close to the prior distribution of the PSGAN, ensuring aesthetic texture quality.
• To achieve efficient, scalable mosaic generation with linear runtime complexity, supporting arbitrarily large output resolutions.
• To overcome limitations of existing style transfer and patch-based methods by optimizing latent space rather than pixel space.

Proposed method

• The method uses a pre-trained Periodic Spatial GAN (PSGAN) as a texture prior to model a rich, multi-texture manifold from multiple example images.
• It formulates a loss function combining a perceptual content loss (using VGG-features) and a novel differentiable texture regularization on local noise channels of the latent tensor Z.
• The optimization is performed over the latent noise tensor Z, not the pixel space, enabling efficient, high-resolution generation.
• The texture regularization enforces statistical independence of local channels in Z, preserving the manifold structure and preventing degenerate textures.
• The generator G is applied to spatially subdivided Z-tensors, allowing parallel computation and memory-efficient processing of very large images.
• The content loss uses a feature-mapping φ (e.g., pooling layers) to downscale the content image, separating low-frequency content from high-frequency texture details.

Experimental results

Research questions

• RQ1 Can optimization in the latent space of a GAN-based texture model produce seamless, high-resolution mosaics that morph naturally between textures?
• RQ2 How can differentiable regularization over the latent noise space preserve texture manifold quality during optimization?
• RQ3 To what extent does using a perceptual loss with feature downsampling improve mosaic quality and frequency separation?
• RQ4 Can the method scale to arbitrarily large image resolutions without memory constraints?
• RQ5 How does GANosaic compare to neural style transfer in preserving texture representation while reconstructing content?

Key findings

• GANosaic generates seamless mosaics with no visible tile borders, achieving high visual quality through smooth texture blending.
• The method supports arbitrarily large output resolutions, limited only by storage, due to spatially decomposable optimization over the latent noise tensor.
• The differentiable texture regularization ensures that optimized noise remains close to the prior distribution, preventing degenerate or unrealistic textures.
• Using a downsampled perceptual loss (via φ) improves mosaic quality by allowing high-frequency content to be reconstructed from the texture manifold.
• The approach outperforms neural style transfer in texture representation, particularly in regions like faces, where style transfer often fails to apply texture and instead mimics content pixels directly.
• Optimization converges to different Z values across spatial regions, enabling local texture variation that adapts to content structure, resulting in a natural mosaic effect.