[Paper Review] CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets

Longwen Zhang, Ziyu Wang+7 more|arXiv (Cornell University)|May 30, 2024

Advanced Data Storage Technologies4 citations

TL;DR

CLAY is a 1.5-billion-parameter, 3D-native generative model that enables high-fidelity, controllable creation of 3D assets using text, images, or 3D primitives via a multi-resolution VAE and latent Diffusion Transformer. It achieves state-of-the-art performance in geometry and appearance quality, outperforming prior methods by 78.9% in geometry and 85.4% in appearance in user studies on text-to-3D and image-to-3D tasks.

ABSTRACT

In the realm of digital creativity, our potential to craft intricate 3D worlds from imagination is often hampered by the limitations of existing digital tools, which demand extensive expertise and efforts. To narrow this disparity, we introduce CLAY, a 3D geometry and material generator designed to effortlessly transform human imagination into intricate 3D digital structures. CLAY supports classic text or image inputs as well as 3D-aware controls from diverse primitives (multi-view images, voxels, bounding boxes, point clouds, implicit representations, etc). At its core is a large-scale generative model composed of a multi-resolution Variational Autoencoder (VAE) and a minimalistic latent Diffusion Transformer (DiT), to extract rich 3D priors directly from a diverse range of 3D geometries. Specifically, it adopts neural fields to represent continuous and complete surfaces and uses a geometry generative module with pure transformer blocks in latent space. We present a progressive training scheme to train CLAY on an ultra large 3D model dataset obtained through a carefully designed processing pipeline, resulting in a 3D native geometry generator with 1.5 billion parameters. For appearance generation, CLAY sets out to produce physically-based rendering (PBR) textures by employing a multi-view material diffusion model that can generate 2K resolution textures with diffuse, roughness, and metallic modalities. We demonstrate using CLAY for a range of controllable 3D asset creations, from sketchy conceptual designs to production ready assets with intricate details. Even first time users can easily use CLAY to bring their vivid 3D imaginations to life, unleashing unlimited creativity.

Motivation & Objective

• To bridge the gap between human imagination and digital 3D creation by enabling effortless, high-fidelity 3D asset generation.
• To overcome limitations in existing 3D generation tools that require extensive expertise and manual labor.
• To develop a scalable, 3D-native generative model capable of handling diverse modalities (text, images, 3D primitives) with high geometric and material fidelity.
• To establish a large-scale, high-quality 3D dataset and training pipeline to support scalable 3D generative modeling.
• To enable both novices and experts to generate complex, production-ready 3D assets with minimal effort.

Proposed method

• CLAY employs a multi-resolution Variational Autoencoder (VAE) to extract hierarchical 3D priors from diverse 3D geometries, enabling efficient latent space representation.
• A minimalistic latent Diffusion Transformer (DiT) is used in the latent space to generate continuous, complete 3D surfaces with high geometric fidelity.
• A progressive training scheme is applied to scale CLAY efficiently on an ultra-large 3D dataset derived from a curated, standardized processing pipeline.
• For appearance generation, CLAY uses a multi-view material diffusion model to generate 2K-resolution PBR textures with diffuse, roughness, and metallic modalities.
• The model supports multi-modal control inputs, including text, images, multi-view images, voxels, bounding boxes, point clouds, and implicit representations.
• A GPT-4V-powered automatic annotation system enhances data efficiency and consistency in the training data pipeline.

Experimental results

Research questions

• RQ1 Can a large-scale, 3D-native generative model achieve high-fidelity geometry and material generation across diverse input modalities?
• RQ2 How does a latent Diffusion Transformer with multi-resolution VAE improve 3D generation quality compared to 2D-based or direct 3D optimization methods?
• RQ3 To what extent can CLAY generalize to complex, composed 3D objects like 'a tiger riding a motorcycle' with text-only inputs?
• RQ4 How does CLAY’s performance compare to state-of-the-art models in both geometry and appearance quality across text-to-3D and image-to-3D tasks?
• RQ5 What impact does a large-scale, curated 3D dataset and progressive training scheme have on model scalability and generation quality?

Key findings

• CLAY achieved 78.9% user preference for geometry quality in text-to-3D generation, significantly outperforming the second-best method, RichDreamer, which required ~2 hours of optimization versus CLAY’s ~45 seconds.
• In image-to-3D generation, CLAY secured 91.2% user preference for geometry quality and 85.4% for appearance quality, demonstrating superior performance and speed.
• CLAY generated 2K-resolution PBR textures with high fidelity across diffuse, roughness, and metallic channels, enabling realistic physically-based rendering.
• The model demonstrated strong generalization to diverse 3D controls, including text, images, and multiple 3D primitive formats, enabling flexible and intuitive design workflows.
• CLAY’s training pipeline, enhanced by GPT-4V for automatic annotation and standardized remeshing, ensured data consistency and scalability across the ultra-large 3D dataset.
• Despite strong performance on single objects, CLAY showed limitations in generating complex composed scenes like 'a tiger riding a motorcycle' due to insufficient training data and descriptive detail in text inputs.