[Paper Review] CAPNet: Continuous Approximation Projection For 3D Point Cloud Reconstruction Using 2D Supervision

K L Navaneet, Priyanka Mandikal+2 more|arXiv (Cornell University)|Nov 28, 2018

Advanced Vision and Imaging20 references4 citations

TL;DR

CAPNet introduces a differentiable, continuous approximation projection module that enables weakly supervised 3D point cloud reconstruction from 2D supervisory signals such as foreground masks. By modeling projections as a smooth, continuous function of 3D points and introducing an affinity loss to suppress outliers, CAPNet achieves state-of-the-art performance on synthetic and real-world datasets, even with only a single mask as supervision, and enables test-time optimization for improved image correspondence.

ABSTRACT

Knowledge of 3D properties of objects is a necessity in order to build effective computer vision systems. However, lack of large scale 3D datasets can be a major constraint for data-driven approaches in learning such properties. We consider the task of single image 3D point cloud reconstruction, and aim to utilize multiple foreground masks as our supervisory data to alleviate the need for large scale 3D datasets. A novel differentiable projection module, called 'CAPNet', is introduced to obtain such 2D masks from a predicted 3D point cloud. The key idea is to model the projections as a continuous approximation of the points in the point cloud. To overcome the challenges of sparse projection maps, we propose a loss formulation termed 'affinity loss' to generate outlier-free reconstructions. We significantly outperform the existing projection based approaches on a large-scale synthetic dataset. We show the utility and generalizability of such a 2D supervised approach through experiments on a real-world dataset, where lack of 3D data can be a serious concern. To further enhance the reconstructions, we also propose a test stage optimization procedure to obtain reconstructions that display high correspondence with the observed input image.

Motivation & Objective

• To address the scarcity of large-scale 3D datasets by enabling 3D point cloud reconstruction using only 2D supervisory signals like foreground masks.
• To overcome the non-differentiability and sparsity issues in conventional point cloud projection methods.
• To improve reconstruction quality by introducing a differentiable, continuous approximation of point cloud projections.
• To develop a novel loss function, affinity loss, that effectively suppresses outlier points in sparse projection maps.
• To demonstrate the generalizability of the approach on real-world datasets with minimal 3D supervision and to enable test-time optimization using observed masks.

Proposed method

• Proposes a continuous approximation projection module that models 2D projections as a differentiable, smooth function of 3D point cloud coordinates using a Gaussian kernel with learnable variance.
• Uses a differentiable rendering process based on kernel density estimation to generate smooth, artifact-free 2D projections from sparse 3D point clouds.
• Introduces an affinity loss that encourages spatial consistency in the projected mask by penalizing disconnected or isolated points, thereby reducing outlier generation.
• Employs a multi-view training setup where multiple 2D masks from different viewpoints are used to supervise the 3D reconstruction process.
• Applies a test-time optimization (TSO) procedure that fine-tunes the predicted point cloud using the input image’s foreground mask to improve geometric and structural correspondence.
• Utilizes a hybrid loss combining binary cross-entropy and affinity loss to balance mask accuracy and structural fidelity in the reconstruction.

Experimental results

Research questions

• RQ1 Can a differentiable, continuous projection module be designed to enable weakly supervised 3D point cloud reconstruction using only 2D supervisory signals like foreground masks?
• RQ2 How can the gradient flow be preserved in sparse projection maps where most regions contain no projected points?
• RQ3 To what extent can a single 2D mask be used to achieve high-quality 3D reconstruction without full 3D supervision?
• RQ4 Can test-time optimization using observed masks significantly improve the correspondence between the reconstructed 3D point cloud and the input image?
• RQ5 How does the proposed affinity loss compare to standard cross-entropy loss in suppressing outlier points and improving reconstruction quality?

Key findings

• CAPNet significantly outperforms existing projection-based methods on a large-scale synthetic dataset, achieving lower Chamfer distances across all categories.
• With only a single foreground mask as supervision, the model achieves competitive performance—Chamfer distance of 2.01 for airplanes and 1.65 for cars on the real-world dataset.
• The test-time optimization (TSO) procedure improves reconstruction quality, with optimized point clouds showing better geometric and structural correspondence to the input image, especially in occluded regions.
• The affinity loss effectively reduces outlier points, as shown by qualitative results and lower Chamfer scores compared to models trained with only binary cross-entropy loss.
• The method generalizes well to real-world data, demonstrating that 2D supervision can be sufficient for high-quality 3D reconstruction when combined with continuous projection and proper loss design.
• The number of views used during training has diminishing returns beyond two projections, with performance stabilizing at 2–3 views, indicating strong sample efficiency.