[Paper Review] Neural Unsigned Distance Fields for Implicit Function Learning

Julian Chibane, Aymen Mir+1 more|arXiv (Cornell University)|Oct 26, 2020

3D Shape Modeling and Analysis80 references31 citations

TL;DR

Introduces Neural Distance Fields (NDF), a neural network representation that predicts unsigned distance fields to surfaces, enabling open surfaces, inner structures, and manifolds; achieves state-of-the-art 3D reconstruction from sparse point clouds and supports dense surface extraction, meshes, and rendering without closing shapes.

ABSTRACT

In this work we target a learnable output representation that allows continuous, high resolution outputs of arbitrary shape. Recent works represent 3D surfaces implicitly with a Neural Network, thereby breaking previous barriers in resolution, and ability to represent diverse topologies. However, neural implicit representations are limited to closed surfaces, which divide the space into inside and outside. Many real world objects such as walls of a scene scanned by a sensor, clothing, or a car with inner structures are not closed. This constitutes a significant barrier, in terms of data pre-processing (objects need to be artificially closed creating artifacts), and the ability to output open surfaces. In this work, we propose Neural Distance Fields (NDF), a neural network based model which predicts the unsigned distance field for arbitrary 3D shapes given sparse point clouds. NDF represent surfaces at high resolutions as prior implicit models, but do not require closed surface data, and significantly broaden the class of representable shapes in the output. NDF allow to extract the surface as very dense point clouds and as meshes. We also show that NDF allow for surface normal calculation and can be rendered using a slight modification of sphere tracing. We find NDF can be used for multi-target regression (multiple outputs for one input) with techniques that have been exclusively used for rendering in graphics. Experiments on ShapeNet show that NDF, while simple, is the state-of-the art, and allows to reconstruct shapes with inner structures, such as the chairs inside a bus. Notably, we show that NDF are not restricted to 3D shapes, and can approximate more general open surfaces such as curves, manifolds, and functions. Code is available for research at https://virtualhumans.mpi-inf.mpg.de/ndf/.

Motivation & Objective

• Motivate a learnable output representation that supports continuous, high-resolution outputs of arbitrary shape.
• Eliminate the requirement to close shapes during training to handle open surfaces and inner structures.
• Enable reconstruction from sparse point clouds into dense point clouds, meshes, and renderable images.
• Demonstrate state-of-the-art reconstruction on ShapeNet and extend NDF to open curves, manifolds, and functions.

Proposed method

• Encode input point clouds with a multi-scale 3D CNN to produce grid features.
• Decode by regressing the unsigned distance to the surface as f(p) = Phi(Psi_x(p)) where Psi_x extracts multi-scale features at p and Phi maps features to a non-negative distance.
• Train with a loss that clamps the distance to a budget delta to focus on the vicinity of the surface.
• Compute analytical gradients of f for surface extraction and normals via back-propagation.
• Provide algorithms to extract dense point clouds (via projection p - f(p)∇f(p)) and to generate meshes with off-the-shelf methods.
• Render images and estimate normals by a modified sphere-tracing procedure.
• Enable multi-target regression by formulating f(x, y) = 0 to model multiple outputs for a given input.

Experimental results

Research questions

• RQ1 Can NDF represent open surfaces and shapes with inner structures, unlike traditional SDF/occupancy-based methods?
• RQ2 Can NDF be trained directly on raw scans without closing the surfaces and still achieve high-quality reconstructions?
• RQ3 Can NDF reconstruct complex real-world scenes and open garments, not just closed 3D models?
• RQ4 Can NDF be extended to represent curves, manifolds, and general functions beyond 3D shapes?
• RQ5 Can surface normals be derived from the NDF gradients to enable rendering and shading?

Key findings

• NDF achieves state-of-the-art reconstruction accuracy on ShapeNet cars from sparse point clouds, outperforming several baselines.
• NDF can reconstruct shapes with inner structures (e.g., chairs inside a bus) and open surfaces, which prior IFL methods cannot represent.
• NDF can represent open curves and manifolds and can interpolate/approximate 2D functions and spirals, demonstrated in function regression experiments.
• Training on raw scans and not requiring closed hulls yields significant accuracy gains over closed-data baselines like SAL.
• The paper provides practical algorithms to obtain dense point clouds, meshes, and renderings directly from the learned unsigned distance field.