[Paper Review] PolyGen: An Autoregressive Generative Model of 3D Meshes

Charlie Nash, Yaroslav Ganin+2 more|arXiv (Cornell University)|Feb 23, 2020

3D Shape Modeling and Analysis29 references65 citations

TL;DR

PolyGen directly models 3D meshes as an autoregressive sequence using a Transformer-based vertex model and a mesh-pointer face model, enabling conditional and unconditional generation of high-quality meshes.

ABSTRACT

Polygon meshes are an efficient representation of 3D geometry, and are of central importance in computer graphics, robotics and games development. Existing learning-based approaches have avoided the challenges of working with 3D meshes, instead using alternative object representations that are more compatible with neural architectures and training approaches. We present an approach which models the mesh directly, predicting mesh vertices and faces sequentially using a Transformer-based architecture. Our model can condition on a range of inputs, including object classes, voxels, and images, and because the model is probabilistic it can produce samples that capture uncertainty in ambiguous scenarios. We show that the model is capable of producing high-quality, usable meshes, and establish log-likelihood benchmarks for the mesh-modelling task. We also evaluate the conditional models on surface reconstruction metrics against alternative methods, and demonstrate competitive performance despite not training directly on this task.

Motivation & Objective

• Motivate direct generative modelling of 3D meshes instead of proxy representations.
• Propose an autoregressive, two-part model (vertex generation followed by face generation) for meshes.
• Enable conditioning on diverse inputs (class, images, voxels) and quantify uncertainty in mesh generation.
• Evaluate using log-likelihood and predictive accuracy, and compare to baselines and alternative mesh representations.

Proposed method

• Two-part autoregressive Mesh Model: vertex model predicts a sequence of quantized vertex coordinates; face model predicts a sequence of vertex indices conditioned on the generated vertices.
• Vertex model uses a Transformer decoder with discrete 8-bit quantized coordinates and a stopping token.
• Face model uses a Transformer plus pointer-network mechanism to produce a variable-length sequence of vertex indices, with masking to enforce valid predictions.
• Optional conditioning context h (class, image, or voxels) is integrated via added vectors for global conditioning or cross-attention to encoders.
• Masking of invalid predictions is applied at evaluation; training uses unmasked likelihood optimization.
• Data augmentation, voxel/image conditioning, and embedding choices are evaluated for their impact on log-likelihood and accuracy.

Experimental results

Research questions

• RQ1 Can a direct mesh generative model produce high-quality, usable meshes without post-processing?
• RQ2 How well can a Transformer-based vertex and a pointer-based face model capture the joint distribution over mesh vertices and faces?
• RQ3 What is the impact of conditioning inputs (class, image, voxels) on vertex/face modelling performance?
• RQ4 How do discrete vertex representations and data augmentation affect modelling efficiency and accuracy?
• RQ5 How does PolyGen perform in unconditional vs conditional mesh generation tasks compared to baselines like Draco and uniform models?

Key findings

• Unconditional PolyGen achieves a best log-likelihood of 4.26 bits per vertex with 85% vertex accuracy and 90% face accuracy.
• Discrete 8-bit vertex embeddings improve bits-per-vertex from 2.56 to 2.46 and overall predictive performance.
• Data augmentation substantially improves performance, reducing bits-per-vertex by ~1.64 on average.
• Cross-attention in the face model degrades performance due to overfitting in their setup.
• Conditional models (class, image, voxels) improve vertex modeling, with voxel conditioning giving the strongest gains; face conditioning provides little or negative gain.