[Paper Review] Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction

Chen-Hsuan Lin, Chen Kong+1 more|arXiv (Cornell University)|Jun 21, 2017

3D Shape Modeling and AnalysisEngineering168 citations

TL;DR

The paper proposes a 2D-convolutional framework to generate dense 3D point clouds for object surfaces, using a pseudo-renderer and multi-view joint optimization instead of traditional 3D ConvNets.

ABSTRACT

Conventional methods of 3D object generative modeling learn volumetric predictions using deep networks with 3D convolutional operations, which are direct analogies to classical 2D ones. However, these methods are computationally wasteful in attempt to predict 3D shapes, where information is rich only on the surfaces. In this paper, we propose a novel 3D generative modeling framework to efficiently generate object shapes in the form of dense point clouds. We use 2D convolutional operations to predict the 3D structure from multiple viewpoints and jointly apply geometric reasoning with 2D projection optimization. We introduce the pseudo-renderer, a differentiable module to approximate the true rendering operation, to synthesize novel depth maps for optimization. Experimental results for single-image 3D object reconstruction tasks show that we outperforms state-of-the-art methods in terms of shape similarity and prediction density.

Motivation & Objective

• Motivate efficient 3D shape generation beyond volumetric voxels by focusing on surface-dense representations.
• Develop a 2D convolutional structure generator to predict 3D points from multiple viewpoints.
• Introduce a differentiable pseudo-renderer to enable depth-map based supervision from novel viewpoints.
• Demonstrate superior shape accuracy and surface density on single-image 3D reconstruction tasks.

Proposed method

• Use a latent encoder to produce a latent representation from input data.
• Predict 3D structure at N viewpoints with a 2D convolutional structure generator producing (x, y, z, mask) per pixel.
• Transform predicted points to canonical 3D coordinates using known viewpoint rotations and intrinsics.
• Introduce a pseudo-renderer to synthesize depth images from novel viewpoints by upsampling projection space and applying max-pooling on inverse depths.
• Train with a joint 2D projection loss comprising mask loss and depth loss on multiple novel viewpoints (L = L_mask + lambda * L_depth).
• Pretrain the structure generator to predict depth images, then fine-tune end-to-end with joint 2D projection optimization.

Experimental results

Research questions

• RQ1 Can dense point clouds generated from 2D convnets match or exceed volumetric/point-based baselines in single-image 3D reconstruction?
• RQ2 Does joint optimization over novel viewpoint projections via a differentiable pseudo-renderer improve surface fidelity and density?
• RQ3 How does explicit factoring of 3D geometry and perspective projection affect reconstruction quality compared to implicit 3D volume learning?

Key findings

• The proposed method yields denser and more accurate shapes than state-of-the-art 3D prediction methods.
• On single-category chair reconstruction, the method achieves lower average 3D distance than all baselines in both directions (pred.→GT and GT→pred).
• Across 13 ShapeNet categories, the method attains lower per-category 3D errors than 3D-R2N2 baselines and often outperforms Fan et al. across single-view predictions.
• Qualitative results show denser point clouds with higher surface fidelity compared to volumetric baselines.
• Latent space manipulations produce smooth interpolations and semantically meaningful shape changes (chairs, tables) indicating interpretable generative structure.