[Paper Review] Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images

Nanyang Wang, Yinda Zhang+4 more|arXiv (Cornell University)|Apr 5, 2018

3D Shape Modeling and Analysis27 references138 citations

TL;DR

Pixel2Mesh generates 3D mesh models from single RGB images using a differentiable pipeline that projects 3D vertices onto the image plane and pools features, outperforming octree-based voxel methods in ShapeNet-car experiments.

ABSTRACT

We propose an end-to-end deep learning architecture that produces a 3D shape in triangular mesh from a single color image. Limited by the nature of deep neural network, previous methods usually represent a 3D shape in volume or point cloud, and it is non-trivial to convert them to the more ready-to-use mesh model. Unlike the existing methods, our network represents 3D mesh in a graph-based convolutional neural network and produces correct geometry by progressively deforming an ellipsoid, leveraging perceptual features extracted from the input image. We adopt a coarse-to-fine strategy to make the whole deformation procedure stable, and define various of mesh related losses to capture properties of different levels to guarantee visually appealing and physically accurate 3D geometry. Extensive experiments show that our method not only qualitatively produces mesh model with better details, but also achieves higher 3D shape estimation accuracy compared to the state-of-the-art.

Motivation & Objective

• Enable 3D mesh reconstruction from single RGB images.
• Integrate differentiable feature pooling and mesh deformation into a learnable pipeline.
• Evaluate against voxel-based baselines and analyze sensitivity to initial meshes.

Proposed method

• Project 3D vertices to the image plane using camera intrinsics and perspective projection.
• Pool image features at projected vertex locations via bilinear interpolation.
• Deform a mesh to fit image-derived features with a Laplacian regularization term for surface smoothness.
• Compare against octree-based voxel reconstruction methods on ShapeNet datasets.
• Analyze sensitivity to initial mesh shapes and provide qualitative visualizations from multiple viewpoints.

Experimental results

Research questions

• RQ1 Can a mesh-based representation with differentiable pooling match or exceed voxel-based reconstructions from a single image?
• RQ2 How does the choice of initial mesh affect reconstruction quality and robustness?
• RQ3 What are the effects of regularization terms (e.g., Laplacian) on mesh quality and smoothness?

Key findings

• The method outperforms the octree-based approach by a margin across F-score and Chamfer-like metrics on ShapeNet-car (F-score 72.128 vs 65.335; F-score 2τ 87.247 vs 79.733; CD 0.236 vs 0.361; EMD 1.220 vs 1.273).
• Hausdorff distance ablation shows the full model generally achieves the best scores, with Laplacian regularization contributing to surface smoothness.
• The approach is not sensitive to the initial mesh shape, with comparable F-scores and CD across sphere and multiple ellipsoid variants.
• Qualitative results demonstrate smooth surfaces and sharp details, and the model generalizes to real-world images.