[Paper Review] Implicit Mesh Reconstruction from Unannotated Image Collections

Shubham Tulsiani, Nilesh Kulkarni+1 more|arXiv (Cornell University)|Jul 16, 2020

3D Shape Modeling and Analysis38 references17 citations

TL;DR

This paper proposes a self-supervised method for implicit mesh reconstruction from unannotated image collections with only foreground masks, using a learned implicit function to deform a sphere into 3D shape while predicting texture and camera pose. Despite relying solely on weak supervision, it achieves competitive performance against methods with stronger supervision, generalizing across 30 diverse object categories with meaningful shape and texture reconstruction.

ABSTRACT

We present an approach to infer the 3D shape, texture, and camera pose for an object from a single RGB image, using only category-level image collections with foreground masks as supervision. We represent the shape as an image-conditioned implicit function that transforms the surface of a sphere to that of the predicted mesh, while additionally predicting the corresponding texture. To derive supervisory signal for learning, we enforce that: a) our predictions when rendered should explain the available image evidence, and b) the inferred 3D structure should be geometrically consistent with learned pixel to surface mappings. We empirically show that our approach improves over prior work that leverages similar supervision, and in fact performs competitively to methods that use stronger supervision. Finally, as our method enables learning with limited supervision, we qualitatively demonstrate its applicability over a set of about 30 object categories.

Motivation & Objective

• To enable 3D shape, texture, and camera pose inference from single RGB images using only category-level image collections with approximate instance segmentations.
• To bypass the need for strong supervision such as 3D annotations, 2D keypoints, or pose labels in 3D reconstruction.
• To learn a category-specific implicit shape space that captures instance-level variations like pose and deformation.
• To derive supervisory signals through geometric consistency and reprojection losses without direct supervision.
• To scale 3D reconstruction to diverse object categories by leveraging only in-the-wild images with weak annotations.

Proposed method

• Represent 3D shape as an image-conditioned implicit function that deforms a sphere into the predicted mesh, with learned texture prediction.
• Enforce consistency between rendered predictions and input images via a differentiable rendering loss.
• Introduce a geometric cycle-consistency loss between local pixel-to-surface mappings and global 3D shape predictions to improve geometric fidelity.
• Use a single 3D template per category as initialization and fine-tune it via end-to-end learning from image collections.
• Optimize with a multi-component loss combining reprojection accuracy, boundary alignment, and rigidity priors to improve generalization.
• Leverage weak supervision from foreground masks and implicitly infer camera pose during training.

Experimental results

Research questions

• RQ1 Can 3D shape, texture, and camera pose be accurately inferred from unannotated image collections with only foreground masks as supervision?
• RQ2 How can geometric consistency between local pixel-to-surface mappings and global 3D predictions be leveraged to improve 3D reconstruction without explicit supervision?
• RQ3 To what extent can implicit shape representations capture instance-level variations such as pose and deformation using only weak supervision?
• RQ4 How does the proposed method compare in performance to prior approaches that use stronger supervision like 2D keypoints or 3D templates?
• RQ5 Can the method generalize across diverse object categories, including those without any annotated 3D data?

Key findings

• The method achieves competitive 3D reconstruction performance on PASCAL3D+ with mean IoU of 0.44 for aeroplane and 0.66 for car, matching or exceeding methods that use stronger supervision.
• On semantic keypoint reprojection, the method achieves 54.1% PCK-P and 41.3% PCK-T, outperforming prior weakly supervised methods.
• Ablation studies show that removing the geometric consistency loss (Lgcc) reduces performance to 50.5% PCK-P, confirming its importance.
• Boundary alignment and rigidity priors significantly improve qualitative results, especially in handling complex shapes and occlusions.
• The model generalizes effectively to 30 diverse categories from ImageNet using only automatically generated masks from PointRend, capturing variations like body shape and head pose.
• Despite no direct supervision, the method produces meaningful textures on visible and invisible regions, demonstrating robust implicit texture learning.