[Paper Review] GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images

Jun Gao, Tianchang Shen+7 more|arXiv (Cornell University)|Sep 22, 2022

Computer Graphics and Visualization Techniques187 citations

TL;DR

GET3D directly generates explicit textured 3D meshes with arbitrary topology from 2D images, using differentiable surface extraction and rendering to achieve state-of-the-art 3D shape and texture quality across multiple categories.

ABSTRACT

As several industries are moving towards modeling massive 3D virtual worlds, the need for content creation tools that can scale in terms of the quantity, quality, and diversity of 3D content is becoming evident. In our work, we aim to train performant 3D generative models that synthesize textured meshes which can be directly consumed by 3D rendering engines, thus immediately usable in downstream applications. Prior works on 3D generative modeling either lack geometric details, are limited in the mesh topology they can produce, typically do not support textures, or utilize neural renderers in the synthesis process, which makes their use in common 3D software non-trivial. In this work, we introduce GET3D, a Generative model that directly generates Explicit Textured 3D meshes with complex topology, rich geometric details, and high-fidelity textures. We bridge recent success in the differentiable surface modeling, differentiable rendering as well as 2D Generative Adversarial Networks to train our model from 2D image collections. GET3D is able to generate high-quality 3D textured meshes, ranging from cars, chairs, animals, motorbikes and human characters to buildings, achieving significant improvements over previous methods.

Motivation & Objective

• Motivate scalable 3D content creation across industries by enabling high-quality, diverse textured meshes with arbitrary topology.
• Enable supervision from 2D image data rather than requiring explicit 3D shapes.
• Produce outputs that are directly usable in standard 3D graphics engines (e.g., Blender, Maya).
• Achieve state-of-the-art quality in geometry and texture across multiple object categories.

Proposed method

• Two-branch generator: a geometry branch outputs a differentiable explicit surface (mesh) with arbitrary topology via DMTet and marching tetrahedra.
• Texture branch outputs a texture field using a tri-plane representation conditioned on geometry and texture latents.
• Differentiable rendering via a high-resolution rasterizer renders RGB images and silhouettes for 2D adversarial supervision.
• Adversarial training with two discriminators (RGB and silhouette) propagates gradients to both geometry and texture branches.
• Cross-entropy regularization on neighboring SDF values to discourage invisible internal faces and improve mesh quality.
• Training uses 2D image supervision with a differentiable renderer to enable high-resolution (up to 1024x1024) textures.

Experimental results

Research questions

• RQ1 Can a single model generate textured meshes with arbitrary topology directly from 2D images without relying on fixed templates?
• RQ2 Does differentiable surface extraction combined with differentiable rendering yield high fidelity geometry and textures when trained with 2D discriminators?
• RQ3 How does GET3D perform on diverse categories with complex geometry and textures compared to 3D-supervised and 3D-aware baselines?
• RQ4 Can the approach be extended to downstream tasks like material generation, relighting, and text-guided 3D synthesis?

Key findings

• GET3D generates high-quality textured meshes with rich geometry and texture detail across multiple categories including cars, chairs, motorbikes, humans, and buildings.
• The model supports arbitrary topology and genus thanks to the DMTet-based surface extraction.
• Training with differentiable rendering and two specialized discriminators improves 3D shape and texture fidelity compared to several baselines.
• Compared with 3D-supervised methods, GET3D achieves better diversity and geometry detail.
• GET3D enables downstream applications such as material generation for view-dependent lighting and text-guided 3D synthesis using CLIP.