[论文解读] Unique3D: High-Quality and Efficient 3D Mesh Generation from a Single Image

Kailu Wu, Fangfu Liu+6 more|arXiv (Cornell University)|May 30, 2024

3D Shape Modeling and Analysis被引用 7

一句话总结

Unique3D 提出一个单图到高保真3D网格的管线，使用多视 diffusion、渐进放大和 ISOMER 网格重建，以实现快速、高质量纹理网格。

ABSTRACT

In this work, we introduce Unique3D, a novel image-to-3D framework for efficiently generating high-quality 3D meshes from single-view images, featuring state-of-the-art generation fidelity and strong generalizability. Previous methods based on Score Distillation Sampling (SDS) can produce diversified 3D results by distilling 3D knowledge from large 2D diffusion models, but they usually suffer from long per-case optimization time with inconsistent issues. Recent works address the problem and generate better 3D results either by finetuning a multi-view diffusion model or training a fast feed-forward model. However, they still lack intricate textures and complex geometries due to inconsistency and limited generated resolution. To simultaneously achieve high fidelity, consistency, and efficiency in single image-to-3D, we propose a novel framework Unique3D that includes a multi-view diffusion model with a corresponding normal diffusion model to generate multi-view images with their normal maps, a multi-level upscale process to progressively improve the resolution of generated orthographic multi-views, as well as an instant and consistent mesh reconstruction algorithm called ISOMER, which fully integrates the color and geometric priors into mesh results. Extensive experiments demonstrate that our Unique3D significantly outperforms other image-to-3D baselines in terms of geometric and textural details.

研究动机与目标

• 提升用单图高效生成高质量3D网格的动机。
• 实现高保真、纹理细节和几何精确度，具强泛化能力。
• 开发一个端到端管线，结合基于扩散的多视合成、法线图和快速网格重构。
• 展示对真实场景单视图图像的实际效率（≤30秒）。

提出的方法

• 使用多视扩散模型和法线扩散模型，生成带法线图的四个正投影多视图图像。
• 逐步将多视 RGB 图像及相应的法线图进行高分辨率（2048）的放大，使用 ControlNet 基于细化和超分步骤。
• 使用 ISOMER（一个即时且一致的网格重构算法）重建高质量网格，将来自 RGB 和法线图的颜色和几何先验整合。
• ISOMER 包含：（a）从前/back 视图的初始网格估计，（b）带可微渲染和基于图像的损失的粗到细优化，（c）明确的优化目标以解决多视不一致和纹理/颜色精细化。

实验结果

研究问题

• RQ1 单图到3D网格的管线是否能在保持多视一致性的同时实现高保真纹理和复杂几何？
• RQ2 在单视监督下，整合专用网格重构（ISOMER）是否比体素/隐式方法在细节和鲁棒性方面有提升？
• RQ3 多级放大策略在不产生过高计算成本的情况下，是否能实现高分辨纹理和准确法线？

主要发现

• Unique3D 相对于现有图像到3D基线在实验中在几何和纹理细节方面有显著提升。
• ISOMER 能在数秒内实现高质量网格重构，端到端输入到网格时间在 RTX4090 上不到 30 秒。
• ISOMER 的重建过程在 10 秒内完成，整个管线实现快速单图到3D结果。
• 基于 GSO 的评估的定量结果显示，我们的方法在 PSNR、SSIM、LPIPS、Clip-Sim、CD、Vol. IoU 和 F-Score 等指标上达到最佳或接近最佳，相对于基线。