[Paper Review] CAT3D: Create Anything in 3D with Multi-View Diffusion Models

Ruiqi Gao, Aleksander Holynski+6 more|arXiv (Cornell University)|May 16, 2024

Simulation Techniques and Applications11 citations

TL;DR

CAT3D introduces a multi-view diffusion model to generate many consistent novel views from limited input views, enabling fast, high-quality 3D reconstruction of scenes from few or even a single image, or text prompts.

ABSTRACT

Advances in 3D reconstruction have enabled high-quality 3D capture, but require a user to collect hundreds to thousands of images to create a 3D scene. We present CAT3D, a method for creating anything in 3D by simulating this real-world capture process with a multi-view diffusion model. Given any number of input images and a set of target novel viewpoints, our model generates highly consistent novel views of a scene. These generated views can be used as input to robust 3D reconstruction techniques to produce 3D representations that can be rendered from any viewpoint in real-time. CAT3D can create entire 3D scenes in as little as one minute, and outperforms existing methods for single image and few-view 3D scene creation. See our project page for results and interactive demos at https://cat3d.github.io .

Motivation & Objective

• Reduce the number of required input views for high-quality 3D scene creation by generating consistent novel views.
• Decouple the generative prior from the 3D reconstruction pipeline to improve efficiency and quality.
• Enable 3D creation from sparse views, a single image, or text prompts with robust 3D reconstruction.

Proposed method

• Train a multi-view diffusion model that conditions on input views and their camera poses to generate multiple target views.
• Use 3D self-attention (spatiotemporal) and raymap camera conditioning to encode per-image pose into the diffusion latent space.
• Generate a large set of views by grouping target viewpoints and autoregressively sampling anchor and subsequent view groups.
• Integrate a robust 3D reconstruction stage (based on Zip-NeRF) with a perceptual loss (LPIPS) and distance-weighted view contributions to handle view inconsistency.
• Fine-tune from a pre-trained latent diffusion model and employ a noise schedule shift proportional to log(N) to accommodate more target views.
• Handle conditioning with nearest-view strategies and manage wide aspect ratios via square-cropped and padded inputs.

Experimental results

Research questions

• RQ1 Can a multi-view diffusion model conditioned on one or more input views produce a large, coherent set of novel views suitable for accurate 3D reconstruction?
• RQ2 Does decoupling generation from 3D reconstruction improve efficiency and quality for few-view and single-image 3D creation?
• RQ3 How do conditioning choices (raymaps, 3D self-attention, number of conditioning views) impact view consistency and final 3D quality?
• RQ4 What is the performance of CAT3D across sparse multi-view, few-view, and single-image to 3D tasks compared to prior work?
• RQ5 What are the limitations and failure modes when reconstructing 3D scenes from generated views versus real photographs?

Key findings

• CAT3D achieves state-of-the-art performance on several few-view 3D reconstruction benchmarks across multiple datasets.
• CAT3D reduces generation time from hours (previous methods) to a few minutes.
• In few-view settings, CAT3D outperforms Zip-NeRF, ZeroNVS, and ReconFusion on PSNR, SSIM, and LPIPS across RealEstate10K, LLFF, DTU, CO3D, and mip-NeRF 360 datasets.
• For single-image to 3D creation, CAT3D provides competitive CLIP-based image scores while offering substantial speed advantages (roughly 1 minute versus up to 120 minutes for some baselines).
• Ablation studies show that 3D self-attention, raymap camera conditioning, and modeling multiple outputs jointly improve both image quality and 3D reconstructions.
• The method maintains plausible content in unseen regions while preserving geometry in observed regions, outperforming several prior approaches in many settings.