[Paper Review] Generation High resolution 3D model from natural language by Generative Adversarial Network

Kentaro Fukamizu, Masaaki Kondo+1 more|arXiv (Cornell University)|Jan 22, 2019

Generative Adversarial Networks and Image Synthesis9 references4 citations

TL;DR

This paper proposes a two-stage conditional Wasserstein GAN framework that generates high-resolution 3D voxel models from natural language descriptions by first producing a low-resolution shape and then refining it into a high-resolution version. The method improves both visual fidelity and text-to-shape faithfulness, achieving a 0.98 classification accuracy and 0.141 MSE on evaluation metrics, outperforming prior low-resolution baselines.

ABSTRACT

We present a method of generating high resolution 3D shapes from natural language descriptions. To achieve this goal, we propose two steps that generating low resolution shapes which roughly reflect texts and generating high resolution shapes which reflect the detail of texts. In a previous paper, the authors have shown a method of generating low resolution shapes. We improve it to generate 3D shapes more faithful to natural language and test the effectiveness of the method. To generate high resolution 3D shapes, we use the framework of Conditional Wasserstein GAN. We propose two roles of Critic separately, which calculate the Wasserstein distance between two probability distribution, so that we achieve generating high quality shapes or acceleration of learning speed of model. To evaluate our approach, we performed quantitive evaluation with several numerical metrics for Critic models. Our method is first to realize the generation of high quality model by propagating text embedding information to high resolution task when generating 3D model.

Motivation & Objective

• To address the challenge of generating high-resolution 3D models from natural language descriptions, which is limited by GPU memory and training time.
• To improve the faithfulness of generated 3D shapes to text descriptions, especially in capturing fine details and color distributions.
• To overcome training instability and slow convergence in high-resolution 3D GANs by redefining the Critic network’s role.
• To develop a scalable, two-stage generative framework that first generates coarse shapes and then refines them into high-resolution outputs.
• To introduce new evaluation metrics—classification accuracy and MSE between text embeddings and encoded voxel features—to assess shape quality and text alignment.

Proposed method

• Uses a two-stage generative process: first generating a low-resolution 3D voxel shape from text, then refining it into a high-resolution version.
• Employs a Conditional Wasserstein GAN with gradient penalty (WGAN-GP) to stabilize training and improve mode coverage.
• Introduces a modified Critic network that incorporates text latent vectors to better assess the Wasserstein distance between real and generated 3D shapes.
• Applies a two-branch encoder to map generated 3D voxels back to 128D text embedding space, enabling MSE-based fidelity evaluation.
• Trains the generator using the Wasserstein loss, where the Critic estimates the distributional distance between real and generated shapes, with text conditioning.
• Uses 64×64×64×4 voxel resolution, significantly higher than prior work’s 32×32×32×4, to improve visual detail and realism.

Experimental results

Research questions

• RQ1 Can a two-stage GAN framework effectively generate high-resolution 3D voxel models from natural language descriptions?
• RQ2 How does incorporating text conditioning into the Critic network affect the fidelity and realism of generated 3D shapes?
• RQ3 Does improving the Critic’s ability to assess distributional similarity lead to more stable and faster training in high-resolution 3D GANs?
• RQ4 To what extent do the proposed evaluation metrics (classification accuracy and MSE) correlate with human perception of shape quality and text alignment?
• RQ5 Can high-resolution 3D generation be achieved without significant degradation in color consistency or structural fidelity compared to low-resolution counterparts?

Key findings

• The v1 model achieved a 0.98 classification accuracy on a 3D shape classifier, outperforming the v0 model’s 0.97, indicating higher realism in generated shapes.
• The v1 model achieved a lower mean squared error (MSE) of 0.141 compared to v0’s 0.156, indicating better alignment between generated voxels and the original text embedding.
• Training loss for the Generator in the v1 model showed lower variance than in v0, suggesting more stable training dynamics.
• The Critic network in v1 demonstrated more consistent and accurate Wasserstein distance estimation by leveraging text-conditioned features.
• Visual inspection confirmed that high-resolution outputs (64×64×64) maintained structural coherence and improved detail compared to low-resolution (32×32×32) baselines.
• The method successfully mitigated shape collapse and preserved color distribution across resolution stages, though minor color shifts were observed.