[Paper Review] One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization

Minghua Liu, Chao Xu+5 more|arXiv (Cornell University)|Jun 29, 2023

3D Shape Modeling and Analysis106 citations

TL;DR

A feed-forward method converts a single image into a full 360° textured 3D mesh in about 45 seconds by combining a view-conditioned 2D diffusion model with a generalizable SDF-based reconstruction, avoiding per-shape optimization.

ABSTRACT

Single image 3D reconstruction is an important but challenging task that requires extensive knowledge of our natural world. Many existing methods solve this problem by optimizing a neural radiance field under the guidance of 2D diffusion models but suffer from lengthy optimization time, 3D inconsistency results, and poor geometry. In this work, we propose a novel method that takes a single image of any object as input and generates a full 360-degree 3D textured mesh in a single feed-forward pass. Given a single image, we first use a view-conditioned 2D diffusion model, Zero123, to generate multi-view images for the input view, and then aim to lift them up to 3D space. Since traditional reconstruction methods struggle with inconsistent multi-view predictions, we build our 3D reconstruction module upon an SDF-based generalizable neural surface reconstruction method and propose several critical training strategies to enable the reconstruction of 360-degree meshes. Without costly optimizations, our method reconstructs 3D shapes in significantly less time than existing methods. Moreover, our method favors better geometry, generates more 3D consistent results, and adheres more closely to the input image. We evaluate our approach on both synthetic data and in-the-wild images and demonstrate its superiority in terms of both mesh quality and runtime. In addition, our approach can seamlessly support the text-to-3D task by integrating with off-the-shelf text-to-image diffusion models.

Motivation & Objective

• Motivate a generic single-image to 3D reconstruction solution that works across object categories.
• Leverage strong 2D diffusion priors to generate multi-view predictions for 3D lifting.
• Develop a 360° mesh reconstruction pipeline that is feed-forward and optimization-free.
• Ensure improved geometry quality and 3D consistency while adhering closely to the input image.

Proposed method

• Use a view-conditioned 2D diffusion model (Zero123) to generate multi-view images from a single input image.
• Estimate the input view elevation and construct camera poses for the multi-view set.
• Apply a cost-volume based, generalizable neural surface reconstruction (SparseNeuS) to produce a textured 3D mesh in a single pass.
• Train with a 2-stage source view selection and groundtruth-prediction mixed supervision to handle inconsistent multi-view predictions.
• Introduce an elevation estimation module to align Zero123 viewpoints with the reconstruction coordinate system.

Figure 1: One-2-3-45 reconstructs a full $360^{\circ}$ mesh of any object in 45 seconds given a single image of it. In each example, we showcase the input image in the left column, alongside the generated textured and textureless meshes from three different views.

Experimental results

Research questions

• RQ1 Can a single image be lifted to a high-quality, textured 360° mesh without per-shape optimization?
• RQ2 How can 2D diffusion priors be effectively leveraged for robust 3D reconstruction across unseen object categories?
• RQ3 What training strategies and pose estimation mechanisms are necessary to reconcile imperfect multi-view predictions with a single forward-pass reconstruction?

Key findings

• The method reconstructs a full 360° textured mesh from a single image in about 45 seconds, without per-shape optimization.
• Using SparseNeuS with a 2-stage view selection and depth supervision yields better 360° geometry and 3D consistency than NeRF/SDF optimization on predicted views.
• Elevation estimation is accurate enough to enable consistent camera poses, critical for correct 3D reconstruction.
• The approach achieves favorable geometry fidelity and adherence to input images compared with competitive zero-shot and optimization-based baselines, while maintaining competitive runtimes.
• The framework can be extended to text-to-3D by integrating with off-the-shelf 2D text-to-image diffusion models.

Figure 2: Our method consists of three primary components: (a) Multi-view synthesis : we use a view-conditioned 2D diffusion model, Zero123 [ 36 ] , to generate multi-view images in a two-stage manner. The input of Zero123 includes a single image and a relative camera transformation, which is parame