[Paper Review] Point-E: A System for Generating 3D Point Clouds from Complex Prompts

Alex Nichol, Heewoo Jun+3 more|arXiv (Cornell University)|Dec 16, 2022

3D Shape Modeling and AnalysisEngineering157 citations

TL;DR

Point E combines a text-to-image diffusion model to render a synthetic view from a prompt and a second diffusion model to generate a colored 3D point cloud conditioned on that view, achieving 1–2 minute single-GPU sampling.

ABSTRACT

While recent work on text-conditional 3D object generation has shown promising results, the state-of-the-art methods typically require multiple GPU-hours to produce a single sample. This is in stark contrast to state-of-the-art generative image models, which produce samples in a number of seconds or minutes. In this paper, we explore an alternative method for 3D object generation which produces 3D models in only 1-2 minutes on a single GPU. Our method first generates a single synthetic view using a text-to-image diffusion model, and then produces a 3D point cloud using a second diffusion model which conditions on the generated image. While our method still falls short of the state-of-the-art in terms of sample quality, it is one to two orders of magnitude faster to sample from, offering a practical trade-off for some use cases. We release our pre-trained point cloud diffusion models, as well as evaluation code and models, at https://github.com/openai/point-e.

Motivation & Objective

• Democratize 3D content creation by reducing sampling time for text-conditioned 3D generation.
• Leverage a two-stage diffusion approach (text-to-image followed by image-to-3D) to handle complex prompts.
• Produce colored 3D point clouds and provide evaluation tools and pre-trained models.
• Scale diffusion-based 3D generation to practical runtimes on a single GPU.

Proposed method

• Generate a synthetic rendered view from a text prompt using a fine-tuned GLIDE model on 3D renderings.
• Create a low-resolution colored point cloud (1,024 points) conditioned on the synthetic view via a transformer-based diffusion model.
• Upsample to a finer colored point cloud (4,096 points) conditioned on the low-resolution cloud and the synthetic view.
• Train on a dataset of several million 3D models converted to 4K colored point clouds from rendered views.
• Condition point cloud diffusion on CLIP-informed image features rather than raw text alone.
• Render meshes from point clouds for evaluation using a regression-based SDF predictor and marching cubes.

Experimental results

Research questions

• RQ1 Can a two-stage diffusion pipeline (text-to-image followed by image-conditioned 3D diffusion) produce coherent colored 3D point clouds from open-ended prompts?
• RQ2 What is the trade-off between sampling speed and final 3D quality relative to state-of-the-art methods?
• RQ3 How does conditioning on richer image representations (grid of CLIP latents) impact 3D generation fidelity and diversity?
• RQ4 What are the limitations and failure modes of image-conditioned 3D diffusion for complex prompts?

Key findings

• Point⋅E can generate diverse and complex colored 3D point clouds conditioned on text prompts.
• Increasing model scale and richer image conditioning improve CLIP R-Precision and P-FID/ P-IS metrics.
• Point⋅E achieves substantially faster sampling (1–2 minutes on a single GPU) than some prior methods, with a trade-off in peak quality.
• Qualitative results show some failure modes due to misinterpretation of object shape or occluded parts in the conditioning image.