[Paper Review] 3DILG: Irregular Latent Grids for 3D Generative Modeling

Biao Zhang, Matthias Nießner+1 more|arXiv (Cornell University)|May 27, 2022

3D Shape Modeling and Analysis22 citations

TL;DR

Introduces irregular latent grids for neural fields to enable scalable, transformer-compatible 3D shape reconstruction and probabilistic generation, achieving state-of-the-art results in reconstruction and various conditioned generation tasks.

ABSTRACT

We propose a new representation for encoding 3D shapes as neural fields. The representation is designed to be compatible with the transformer architecture and to benefit both shape reconstruction and shape generation. Existing works on neural fields are grid-based representations with latents defined on a regular grid. In contrast, we define latents on irregular grids, enabling our representation to be sparse and adaptive. In the context of shape reconstruction from point clouds, our shape representation built on irregular grids improves upon grid-based methods in terms of reconstruction accuracy. For shape generation, our representation promotes high-quality shape generation using auto-regressive probabilistic models. We show different applications that improve over the current state of the art. First, we show results for probabilistic shape reconstruction from a single higher resolution image. Second, we train a probabilistic model conditioned on very low resolution images. Third, we apply our model to category-conditioned generation. All probabilistic experiments confirm that we are able to generate detailed and high quality shapes to yield the new state of the art in generative 3D shape modeling.

Motivation & Objective

• Motivate a latent representation for 3D shapes that is compatible with transformer architectures and supports both reconstruction and generation.
• Develop irregular latent grids that are sparse and adaptive, avoiding the limitations of regular grids.
• Demonstrate improved 3D shape reconstruction from point clouds and state-of-the-art probabilistic 3D shape generation across multiple conditioning regimes.

Proposed method

• Define a fixed-length sequence of latent tuples (x_i, z_i) where x_i are 3D positions and z_i are latents on irregular grids.
• Process patches of point neighborhoods via a mini-PointNet-like embedding to produce i-th patch embedding e_i.
• Use a transformer on the sequence of (e_i, position embedding p_i) to learn local latents z_i.
• Optionally apply vector quantization with a dictionary D to discretize intermediate latents for autoregressive modeling.
• Interpolate latents z_x for arbitrary query x using an autoregressive- or kernel-based Nadarya-Watson estimator and decode via an MLP to obtain an occupancy probability O(x).
• Provide autoregressive and bidirectional transformer strategies to generate z_i conditioned on coordinates or other conditioning signals; support unidirectional token-by-token generation or block-wise bidirectional sampling.

Experimental results

Research questions

• RQ1 Can irregular latent grids match or surpass grid-based representations in reconstruction accuracy from point clouds?
• RQ2 Do irregular latent grids enable high-quality probabilistic 3D shape generation conditioned on images, category labels, or point clouds?
• RQ3 How does vector quantization affect quality and tractability in 3DILG’s autoregressive generation?
• RQ4 Can the model support multiple conditioning modalities (image, category, point cloud) with a fixed-length latent sequence and transformer-based modeling?

Key findings

• Irregular latent grids yield competitive to state-of-the-art reconstruction results on ShapeNet-v2 across multiple metrics.
• The model supports probabilistic, multi-sample 3D shape generation conditioned on high- or low-resolution images, category labels, or point clouds, achieving high-quality surface details.
• Vector quantization can slightly reduce reconstruction/generation performance but provides a discrete latent space beneficial for autoregressive modeling.
• Category-conditioned generation using irregular latents yields favorable FID scores compared to a grid-based 8^3 baseline, demonstrating improved detail and diversity in generated shapes.