[Paper Review] Era3D: High-Resolution Multiview Diffusion using Efficient Row-wise Attention

Peng Li, Yuan Liu+11 more|arXiv (Cornell University)|May 19, 2024

Advanced Image Processing Techniques4 citations

TL;DR

Era3D proposes a high-resolution multiview diffusion model that generates 512×512 multiview images from single-view inputs by introducing a diffusion-based camera prediction module to estimate focal length and elevation, and a row-wise attention mechanism to enforce epipolar priors. This reduces computational complexity by 12x and enables high-fidelity 3D mesh reconstruction, outperforming prior methods in resolution and robustness to camera intrinsics mismatch.

ABSTRACT

In this paper, we introduce Era3D, a novel multiview diffusion method that generates high-resolution multiview images from a single-view image. Despite significant advancements in multiview generation, existing methods still suffer from camera prior mismatch, inefficacy, and low resolution, resulting in poor-quality multiview images. Specifically, these methods assume that the input images should comply with a predefined camera type, e.g. a perspective camera with a fixed focal length, leading to distorted shapes when the assumption fails. Moreover, the full-image or dense multiview attention they employ leads to an exponential explosion of computational complexity as image resolution increases, resulting in prohibitively expensive training costs. To bridge the gap between assumption and reality, Era3D first proposes a diffusion-based camera prediction module to estimate the focal length and elevation of the input image, which allows our method to generate images without shape distortions. Furthermore, a simple but efficient attention layer, named row-wise attention, is used to enforce epipolar priors in the multiview diffusion, facilitating efficient cross-view information fusion. Consequently, compared with state-of-the-art methods, Era3D generates high-quality multiview images with up to a 512*512 resolution while reducing computation complexity by 12x times. Comprehensive experiments demonstrate that Era3D can reconstruct high-quality and detailed 3D meshes from diverse single-view input images, significantly outperforming baseline multiview diffusion methods. Project page: https://penghtyx.github.io/Era3D/.

Motivation & Objective

• To address the limitation of existing multiview diffusion models that assume fixed camera types, leading to shape distortions when input images deviate from predefined intrinsics.
• To reduce the computational complexity of dense multiview attention, which scales poorly with resolution and hinders high-resolution generation.
• To enable high-resolution (512×512) multiview image generation for improved 3D reconstruction fidelity.
• To improve robustness to arbitrary camera intrinsics and viewpoints by learning camera parameters end-to-end from input images.
• To facilitate accurate 3D mesh reconstruction from generated multiview images by ensuring multiview consistency and geometric accuracy.

Proposed method

• A diffusion-based camera prediction module estimates the focal length and elevation of the input image, enabling adaptation to arbitrary camera intrinsics without shape distortion.
• A novel row-wise attention mechanism enforces epipolar constraints by attending only along corresponding rows across views, reducing computation from O(N²S⁴) to O(N²S³).
• The method operates in a canonical camera setting with orthogonal cameras and fixed elevation (0°), simplifying epipolar geometry and enabling efficient attention computation.
• The model uses a denoising U-Net with cross-attention between views, where attention is computed only along row-aligned epipolar lines.
• Camera prediction is supervised using ground-truth pose during training, with regularization to prevent error accumulation across denoising steps.
• The framework supports training with randomly dropped condition images, improving generalization and robustness.

Figure 1 : Given single-view image with arbitrary intrinsic and viewpoints, Era3D can generate high-quality multiview images with a resolution of $512\times 512$ on the orthogonal camera setting, which can be used in mesh reconstruction by NeuS [ 102 ] .

Experimental results

Research questions

• RQ1 Can a diffusion model accurately predict camera intrinsics (focal length and elevation) from a single-view input to enable distortion-free multiview generation?
• RQ2 How can multiview attention be made computationally efficient at high resolutions without sacrificing cross-view consistency?
• RQ3 To what extent can row-wise attention preserve epipolar geometry and improve generation quality compared to dense multiview attention?
• RQ4 Can the proposed method generate high-resolution (512×512) multiview images that support accurate 3D mesh reconstruction?
• RQ5 How robust is the method to variations in input camera parameters such as focal length and viewpoint?

Key findings

• Era3D achieves a 12x reduction in computational complexity compared to state-of-the-art multiview diffusion models by replacing dense multiview attention with row-wise attention.
• The model generates high-resolution multiview images at 512×512 resolution, significantly improving upon the 256×256 limit of prior methods.
• On the GSO dataset, the average elevation prediction error is 0.29°, with a variance of only 32.1, demonstrating high accuracy and robustness.
• Focal length prediction error is reduced to 0.01 (normalized) on average, with lower variance than baselines, showing strong generalization to different focal lengths.
• Chamfer Distance (CD) for reconstructed meshes remains below 0.022 across diverse camera settings, including extreme focal lengths (e.g., f=135mm) and elevations (e.g., α=40°), indicating strong geometric fidelity.
• Visual results show consistent and distortion-free multiview generation across diverse inputs, including real-world internet images, with accurate normal maps and mesh reconstructions.

Figure 2 : Different types of multiview attention layers. ( a ) In a dense multiview attention layer, all feature vectors of multiview images are fed into an attention block. For a general camera setting ( b ) with arbitrary viewpoints and intrinsics, utilizing epipolar constraint to construct an ep