[Paper Review] Learning to Infer Implicit Surfaces without 3D Supervision

Shichen Liu, Shunsuke Saito+2 more|arXiv (Cornell University)|Nov 2, 2019

3D Shape Modeling and Analysis89 citations

TL;DR

Introduces unsupervised learning of implicit 3D surfaces from single or multi-view 2D images using a ray-based field probing method and geometric regularization, achieving high-fidelity topology-aware shapes without 3D supervision.

ABSTRACT

Recent advances in 3D deep learning have shown that it is possible to train highly effective deep models for 3D shape generation, directly from 2D images. This is particularly interesting since the availability of 3D models is still limited compared to the massive amount of accessible 2D images, which is invaluable for training. The representation of 3D surfaces itself is a key factor for the quality and resolution of the 3D output. While explicit representations, such as point clouds and voxels, can span a wide range of shape variations, their resolutions are often limited. Mesh-based representations are more efficient but are limited by their ability to handle varying topologies. Implicit surfaces, however, can robustly handle complex shapes, topologies, and also provide flexible resolution control. We address the fundamental problem of learning implicit surfaces for shape inference without the need of 3D supervision. Despite their advantages, it remains nontrivial to (1) formulate a differentiable connection between implicit surfaces and their 2D renderings, which is needed for image-based supervision; and (2) ensure precise geometric properties and control, such as local smoothness. In particular, sampling implicit surfaces densely is also known to be a computationally demanding and very slow operation. To this end, we propose a novel ray-based field probing technique for efficient image-to-field supervision, as well as a general geometric regularizer for implicit surfaces, which provides natural shape priors in unconstrained regions. We demonstrate the effectiveness of our framework on the task of single-view image-based 3D shape digitization and show how we outperform state-of-the-art techniques both quantitatively and qualitatively.

Motivation & Objective

• Motivate learning high-quality 3D shapes from abundant 2D images without 3D ground-truth supervision.
• Leverage implicit surface representations to handle arbitrary topology and multi-resolution details.
• Develop efficient image-to-field supervision via a ray-based probing framework.
• Introduce geometric regularization to enforce local surface smoothness and plausible normals.

Proposed method

• Define occupancy probability by a neural implicit field decoded from image features.
• Use sparse 3D anchor points with spherical regions to probe the implicit field.
• Cast rays through image pixels and aggregate occupancy from intersecting anchors via max pooling.
• Apply boundary-aware assignment to reduce labeling ambiguities near the surface boundary.
• Employ importance sampling to concentrate samples around image silhouettes and 3D surfaces.
• Introduce a finite-difference based geometric regularizer weighted to emphasize surface regions.

Experimental results

Research questions

• RQ1 Can implicit surfaces be learned from 2D supervision without 3D ground-truth data?
• RQ2 How can we efficiently couple 2D image cues with an implicit 3D field to infer accurate, topologically complex shapes?
• RQ3 What regularization strategies best enforce plausible geometry on implicit surfaces trained from 2D silhouettes?

Key findings

• Achieves state-of-the-art 3D IoU among unsupervised methods on ShapeNet using 2D silhouettes.
• Produces higher-resolution, topology-flexible reconstructions compared to voxels, point clouds, and meshes.
• Demonstrates that the proposed field probing with anchors and rays improves accuracy and detail.
• Geometric regularization via finite differences enables controllable surface smoothness and normals through the p-norm.
• Ablation studies confirm the benefits of boundary-aware assignment and importance sampling for quality gains.