[Paper Review] DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction

Qiangeng Xu, Weiyue Wang+3 more|arXiv (Cornell University)|May 26, 2019

Advanced Vision and Imaging33 references239 citations

TL;DR

DISN predicts continuous signed distance fields from a single image by combining global and local image features, enabling high-quality, detail-rich 3D mesh reconstructions without fixed topology. It outperforms state-of-the-art single-view methods and supports multi-view extensions.

ABSTRACT

Reconstructing 3D shapes from single-view images has been a long-standing research problem. In this paper, we present DISN, a Deep Implicit Surface Network which can generate a high-quality detail-rich 3D mesh from an 2D image by predicting the underlying signed distance fields. In addition to utilizing global image features, DISN predicts the projected location for each 3D point on the 2D image, and extracts local features from the image feature maps. Combining global and local features significantly improves the accuracy of the signed distance field prediction, especially for the detail-rich areas. To the best of our knowledge, DISN is the first method that constantly captures details such as holes and thin structures present in 3D shapes from single-view images. DISN achieves the state-of-the-art single-view reconstruction performance on a variety of shape categories reconstructed from both synthetic and real images. Code is available at https://github.com/xharlie/DISN The supplementary can be found at https://xharlie.github.io/images/neurips_2019_supp.pdf

Motivation & Objective

• Motivate improving single-view 3D reconstruction with implicit surfaces to capture fine details and holes.
• Propose a Deep Implicit Surface Network (DISN) that predicts SDF values from 2D images.
• Incorporate a local feature extraction module by projecting 3D query points into the image to gather patch-based features.
• Demonstrate state-of-the-art accuracy on ShapeNet Core across multiple categories and analyze the impact of camera pose and local features.
• Showcase extensions to multi-view reconstruction and related applications.

Proposed method

• Represent 3D shapes as signed distance functions (SDF) and predict SDF values for 3D points given an input image.
• Use a CNN encoder to extract global image features and estimate a 6D camera pose representation to project 3D points onto the image plane for local feature extraction.
• Augment SDF prediction with a local feature stream extracted from feature maps at the projected image location, concatenated with the global features and point features.
• Train with a regression loss on continuous SDF values, including a weighted loss to emphasize regions near the iso-surface for detail fidelity.
• Reconstruct surfaces by sampling a dense 3D grid, predicting SDF values, and extracting a mesh via Marching Cubes.
• Extend to multi-view reconstruction by aggregating features from multiple views (via max pooling) before decoding.

Experimental results

Research questions

• RQ1 Can a deep network predict continuous SDF values for arbitrary 3D points from a single image without fixed topology constraints?
• RQ2 Does incorporating local image features (via projected patches) improve reconstruction of fine details such as holes and thin structures over using global features alone?
• RQ3 How does estimated camera pose affect SDF prediction and final mesh quality in single-view reconstruction?
• RQ4 Can the approach be extended to multi-view inputs to further improve detail and accuracy?
• RQ5 How does DISN compare to state-of-the-art methods on synthetic and real data in terms of standard 3D reconstruction metrics?

Key findings

• DISN achieves state-of-the-art performance on single-view 3D reconstruction across ShapeNet Core categories, outperforming baselines in EMD and IoU metrics.
• Incorporating a local feature extraction module (via projected image patches) significantly improves recovery of fine-grained details compared to using only global features.
• The 6D rotation representation for camera pose and the proposed loss facilitate stable training and accurate pose estimation, contributing to improved reconstruction quality.
• Ground-truth camera poses yield better quantitative results, but estimated poses still produce competitive qualitative reconstructions with small reprojection errors.
• DISN demonstrates robustness to camera pose estimation errors, and its SDF-based framework yields smoother surfaces than fixed-volume 3D CNN approaches.
• The approach generalizes to real images and online product images, and supports multi-view reconstruction with further improvements from multi-view feature aggregation.