[Paper Review] LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation

Jiaxiang Tang, Zhaoxi Chen+4 more|arXiv (Cornell University)|Feb 7, 2024

Computer Graphics and Visualization Techniques4 citations

TL;DR

LGM proposes a large multi-view Gaussian model that generates high-resolution 3D content in under 5 seconds from text or single-view images, using an asymmetric U-Net to predict 3D Gaussians from multi-view images and fusing them via differentiable rendering. The method achieves 512×512 training resolution with up to 65,536 Gaussians, significantly improving resolution and speed over prior feed-forward 3D generation models.

ABSTRACT

3D content creation has achieved significant progress in terms of both quality and speed. Although current feed-forward models can produce 3D objects in seconds, their resolution is constrained by the intensive computation required during training. In this paper, we introduce Large Multi-View Gaussian Model (LGM), a novel framework designed to generate high-resolution 3D models from text prompts or single-view images. Our key insights are two-fold: 1) 3D Representation: We propose multi-view Gaussian features as an efficient yet powerful representation, which can then be fused together for differentiable rendering. 2) 3D Backbone: We present an asymmetric U-Net as a high-throughput backbone operating on multi-view images, which can be produced from text or single-view image input by leveraging multi-view diffusion models. Extensive experiments demonstrate the high fidelity and efficiency of our approach. Notably, we maintain the fast speed to generate 3D objects within 5 seconds while boosting the training resolution to 512, thereby achieving high-resolution 3D content generation.

Motivation & Objective

• To address the limitation of low training resolution and high computational cost in existing 3D generation models.
• To overcome the inefficiency of triplane-based representations and transformer backbones in high-resolution 3D generation.
• To enable fast, high-fidelity 3D generation from text or single-view images using a feed-forward, end-to-end trainable framework.
• To improve robustness to domain gaps between real and synthesized multi-view images through data augmentation.
• To provide a general mesh extraction pipeline for generated 3D Gaussians for downstream applications.

Proposed method

• The method uses multi-view Gaussian features as an expressive and efficient 3D representation, enabling compact scene representation and fast rendering.
• An asymmetric U-Net backbone processes multi-view images to predict 3D Gaussian features, with attention blocks in deeper layers to enhance cross-view feature fusion.
• Differentiable rendering is applied to supervise the model using high-resolution novel views, enabling end-to-end training with image-level supervision.
• The model leverages pre-trained text-to-image or image-to-multi-view diffusion models to generate four input views from text or a single image.
• Two data augmentation strategies are introduced to bridge the domain gap between real 3D-rendered views and diffusion-generated multi-view images.
• A general mesh extraction algorithm converts the generated 3D Gaussians into smooth, textured polygonal meshes for downstream use.

Experimental results

Research questions

• RQ1 Can a feed-forward 3D generation model achieve high-resolution output (512×512) while maintaining sub-5-second inference speed?
• RQ2 How effective is multi-view Gaussian fusion with an asymmetric U-Net backbone in preserving geometric and textural details compared to triplane or NeRF-based methods?
• RQ3 To what extent do data augmentations improve robustness when training on synthetic multi-view images from diffusion models?
• RQ4 Can the proposed method generalize across both text-to-3D and image-to-3D generation tasks with high fidelity?
• RQ5 How well can 3D Gaussians be converted into high-quality, textured meshes suitable for real-world applications?

Key findings

• The LGM model generates 3D Gaussians with up to 65,536 points and supports 512×512 training resolution, significantly increasing the resolution limit of feed-forward 3D generation.
• The model achieves 3D content generation in approximately 5 seconds, maintaining high speed while boosting resolution beyond prior feed-forward models.
• User studies show that LGM-generated 3D Gaussians are preferred over baselines in image consistency and overall quality, with 600 valid scores collected from 20 volunteers.
• Ablation studies confirm that multi-view input improves 3D geometry reconstruction, with single-view models producing blurry and inconsistent back views.
• Data augmentation significantly reduces floaters and geometric artifacts, improving 3D consistency despite domain gaps in multi-view image synthesis.
• The mesh extraction algorithm successfully converts 3D Gaussians into smooth, textured meshes, enabling compatibility with standard 3D pipelines.