[Paper Review] LRM: Large Reconstruction Model for Single Image to 3D

Yicong Hong, Kai Zhang+8 more|arXiv (Cornell University)|Nov 8, 2023

3D Shape Modeling and AnalysisEngineering30 citations

TL;DR

LRM is the first large transformer-based model that reconstructs a 3D NeRF from a single image in about five seconds, trained on roughly one million 3D shapes and video data to generalize to in-the-wild inputs.

ABSTRACT

We propose the first Large Reconstruction Model (LRM) that predicts the 3D model of an object from a single input image within just 5 seconds. In contrast to many previous methods that are trained on small-scale datasets such as ShapeNet in a category-specific fashion, LRM adopts a highly scalable transformer-based architecture with 500 million learnable parameters to directly predict a neural radiance field (NeRF) from the input image. We train our model in an end-to-end manner on massive multi-view data containing around 1 million objects, including both synthetic renderings from Objaverse and real captures from MVImgNet. This combination of a high-capacity model and large-scale training data empowers our model to be highly generalizable and produce high-quality 3D reconstructions from various testing inputs, including real-world in-the-wild captures and images created by generative models. Video demos and interactable 3D meshes can be found on our LRM project webpage: https://yiconghong.me/LRM.

Motivation & Objective

• Motivate a generic, fast single-image to 3D reconstruction that works across diverse object categories without heavy per-shape optimization.
• Leverage large-scale data and a high-capacity transformer to learn a robust 3D prior.
• Directly predict a NeRF in triplane form from a 2D image using cross-attention and self-attention within a transformer framework.

Proposed method

• Use a pre-trained DINO ViT encoder to extract patch-wise image features.
• Introduce an image-to-triplane transformer decoder that projects 2D image features onto a 3D triplane via cross-attention and refines via self-attention.
• Represent 3D with a triplane NeRF: three orthogonal planes (XY, YZ, XZ) with learnable features that are queried by 3D points and decoded by an MLP to color and density.
• Condition transformer layers with camera features through ModLN (adaptive layer normalization) to guide triplane feature generation.
• Train end-to-end on ~1M 3D shapes and videos using simple image reconstruction losses across novel views (MSE + LPIPS).
• During inference, render a 3D mesh from the triplane-NeRF in under 5 seconds on an NVIDIA A100.

Experimental results

Research questions

• RQ1 Can a single, large-capacity transformer model learn a generic 3D prior to reconstruct arbitrary objects from a single image?
• RQ2 How well does a data-driven, multi-view-supervised approach generalize to real-world and generated inputs without per-object optimization?
• RQ3 What is the effectiveness of a triplane NeRF representation when driven by cross-attention from 2D image features?
• RQ4 How does camera conditioning influence convergence and reconstruction quality in single-image 3D reconstruction?

Key findings

• LRM achieves high-fidelity 3D reconstructions from single images across real, generated, and rendered inputs.
• The model can reconstruct shapes from diverse categories with a 5-second inference time on a single A100 GPU.
• LRM uses a 500M-parameter transformer to map 2D image features to a 3D triplane NeRF representation with end-to-end training.
• A triplane NeRF provides a compact, efficient 3D representation suitable for fast rendering and detailed geometry/color capture.
• Training on approximately 1 million 3D shapes and videos enables generalization beyond category-specific priors.
• LRM can extract meshes via Marching Cubes from the triplane-NeRF after querying a dense point grid.