[论文解读] Consistent123: Improve Consistency for One Image to 3D Object Synthesis

Haohan Weng, Tianyu Yang+5 more|arXiv (Cornell University)|Oct 12, 2023

Generative Adversarial Networks and Image Synthesis被引用 10

一句话总结

Consistent123 同时生成多视图，利用跨视图注意力和共享自注意力，以在图像到3D对象合成中实现强视图一致性，并在采样阶段采用渐进的无分类器引导策略。

ABSTRACT

Large image diffusion models enable novel view synthesis with high quality and excellent zero-shot capability. However, such models based on image-to-image translation have no guarantee of view consistency, limiting the performance for downstream tasks like 3D reconstruction and image-to-3D generation. To empower consistency, we propose Consistent123 to synthesize novel views simultaneously by incorporating additional cross-view attention layers and the shared self-attention mechanism. The proposed attention mechanism improves the interaction across all synthesized views, as well as the alignment between the condition view and novel views. In the sampling stage, such architecture supports simultaneously generating an arbitrary number of views while training at a fixed length. We also introduce a progressive classifier-free guidance strategy to achieve the trade-off between texture and geometry for synthesized object views. Qualitative and quantitative experiments show that Consistent123 outperforms baselines in view consistency by a large margin. Furthermore, we demonstrate a significant improvement of Consistent123 on varying downstream tasks, showing its great potential in the 3D generation field. The project page is available at consistent-123.github.io.

研究动机与目标

• 推动从单个有条件图像进行几何信息自由的3D对象合成时的视图一致性提升。
• 提出一种架构，使得在跨视图交互下能够实现同时多视图生成。
• 通过利用预训练权重和最少的可训练参数来保持零样本能力。
• 引入允许任意数量视图的采样策略，并优化几何与纹理之间的权衡。

提出的方法

• 在去噪U-Net的每个自注意力层之后加入跨视图注意力，以实现合成视图之间的交互。
• 应用一个共享自注意力机制，使所有视图对条件输入视图进行注意，从而更好地对齐空间布局。
• 在将输入视图与多个带噪声的视图级联并以输入视图的 CLIP 嵌入和相对姿态变换为条件进行训练，在与输入视图拼接的多个带噪声的视图上进行训练。
• 在采样阶段实现可同时生成任意数量的视图，训练设置为固定长度视图，但采样时为任意长度视图。
• 引入渐进式无分类器引导（PCFG），在去噪过程中逐步降低引导强度，平衡几何与纹理细节。
• 从预训练的 Zero123 模型初始化空间层，同时保持跨视图注意力权重可训练，以保持零样本能力。

Figure 1: Given the input view and relative pose sequence, Consistent123 can synthesize consistent novel views concurrently, while Zero123 fails at producing consistent views.

实验结果

研究问题

• RQ1 在任意对象类别上，同时进行多视图扩散生成是否能够提升视图一致性？
• RQ2 跨视图注意力和共享自注意力如何影响条件视图与合成视图之间的对齐？
• RQ3 哪些采样策略（任意长度视图、PCFG）在实践中能优化几何-纹理权衡和一致性？
• RQ4 与基线相比，Consistent123 在下游任务如3D重建和图像到3D生成中的表现如何？

主要发现

数据集	模型	PSNR ↑	SSIM ↑	LPIPS ↓
Objaverse Testset	Zero123	21.72	0.92	0.23
Objaverse Testset	Zero123 + SC	22.09	0.92	0.21
Objaverse Testset	Consistent123	24.98	0.96	0.14
GSO	Zero123	22.88	0.92	0.25
GSO	Zero123 + SC	22.30	0.93	0.21
GSO	Consistent123	27.98	0.98	0.11
RTMV	Zero123	15.68	0.78	0.36
RTMV	Zero123 + SC	15.88	0.76	0.36
RTMV	Consistent123	18.76	0.85	0.25

• Consistent123 在多个基准上显著提升了相较于 Zero123 与带随机条件的 Zero123 的视图一致性。
• 在 Objaverse、GSO 和 RTMV 数据集上，使用跨视图注意力和共享自注意力训练可获得更高的 PSNR 和 SSIM，以及更低的 LPIPS。
• 消融研究表明，跨视图注意力是实现一致性的最关键组件，共享自注意力和 PCFG 提供额外增益。
• 在采样阶段同时生成更多视图提升一致性和质量，这与自回归的预期相反。
• PCFG（尤其是 concave reduction）在几何-纹理平衡方面表现更好，减少伪影同时保留细节。

Figure 2: The overall method of Consistent123. (a) At the training stage, multiple noisy views concatenated (denoted as $\oplus$ ) with the input view are fed into the denoising U-Net simultaneously, conditioned on the CLIP embedding of the input view and the corresponding poses. For sampling, views