[Paper Review] Solid Texture Synthesis using Generative Adversarial Networks

Xin Zhao, Lin Wang+4 more|arXiv (Cornell University)|Feb 8, 2021

Generative Adversarial Networks and Image Synthesis37 references4 citations

TL;DR

This paper proposes STS-GAN, a GAN-based method for solid texture synthesis that learns 2D exemplar distributions through volumetric operations in a feature-free manner. By employing multi-scale discriminators to compare patch exemplars with generated volume slices, the model produces high-quality, realistic 3D textures with strong visual fidelity to the input exemplar.

ABSTRACT

Solid texture synthesis, as an effective way to extend 2D exemplar to a volumetric texture, exhibits advantages in numerous application domains. However, existing methods generally suffer from synthesis distortion due to the under-utilization of information. In this paper, we propose a novel approach for the solid texture synthesis based on generative adversarial networks(GANs), named STS-GAN, learning the distribution of 2D exemplars with volumetric operation in a feature-free manner. The multi-scale discriminators evaluate the similarities between patch exemplars and slices from generated volume, promoting the generator to synthesize realistic solid texture. Experimental results demonstrate that the proposed method can synthesize high-quality solid texture with similar visual characteristics to the exemplar.

Motivation & Objective

• Address the limitation of existing solid texture synthesis methods that suffer from synthesis distortion due to under-utilization of exemplar information.
• Develop a method that effectively extends 2D texture exemplars into high-quality 3D volumetric textures without relying on hand-crafted features.
• Improve realism and structural consistency in synthesized solid textures by leveraging adversarial training with multi-scale discriminators.
• Enable feature-free learning of 2D exemplar distributions to preserve visual characteristics in 3D generation.

Proposed method

• Propose STS-GAN, a generative adversarial network architecture specifically designed for solid texture synthesis from 2D exemplars.
• Use volumetric operations to transform 2D exemplars into 3D textures during the generation process, enabling spatial consistency.
• Implement multi-scale discriminators that evaluate local patch similarities between exemplar patches and slices extracted from the generated volume.
• Train the generator adversarially to fool the discriminators by producing slices that are indistinguishable from real exemplar patches at multiple scales.
• Employ a feature-free learning strategy to directly model the distribution of 2D exemplars without requiring intermediate feature representations.
• Optimize the generator and discriminators end-to-end using adversarial loss to enhance realism and structural fidelity.

Experimental results

Research questions

• RQ1 Can a GAN-based approach effectively synthesize high-quality 3D solid textures from 2D exemplars without relying on hand-crafted features?
• RQ2 How does multi-scale discrimination improve the realism and consistency of synthesized volumetric textures?
• RQ3 To what extent does the volumetric operation preserve the visual characteristics of the original 2D exemplar in 3D space?
• RQ4 Does the feature-free learning strategy outperform methods that depend on intermediate feature engineering in solid texture synthesis?

Key findings

• The proposed STS-GAN method successfully synthesizes 3D textures with high visual quality and strong resemblance to the input 2D exemplars.
• The use of multi-scale discriminators significantly improves the realism of generated texture slices by enforcing local patch-level consistency.
• The feature-free learning approach enables effective distribution modeling of 2D exemplars, reducing distortion in the synthesized volumes.
• Experimental results confirm that STS-GAN outperforms existing methods in preserving visual characteristics and minimizing synthesis artifacts.