[Paper Review] Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer

Felix Petersen, Amit H. Bermano+2 more|arXiv (Cornell University)|Mar 26, 2019

Advanced Vision and Imaging39 references41 citations

TL;DR

Pix2Vex learns to reconstruct 3D geometry from images by pairing a novel smooth differentiable renderer with image-to-image translators in a reconstructive adversarial network, enabling 3D prediction with minimal supervision and no ground-truth 3D models.

ABSTRACT

The long-coveted task of reconstructing 3D geometry from images is still a standing problem. In this paper, we build on the power of neural networks and introduce Pix2Vex, a network trained to convert camera-captured images into 3D geometry. We present a novel differentiable renderer ($DR$) as a forward validation means during training. Our key insight is that $DR$s produce images of a particular appearance, different from typical input images. Hence, we propose adding an image-to-image translation component, converting between these rendering styles. This translation closes the training loop, while allowing to use minimal supervision only, without needing any 3D model as ground truth. Unlike state-of-the-art methods, our $DR$ is $C^\infty$ smooth and thus does not display any discontinuities at occlusions or dis-occlusions. Through our novel training scheme, our network can train on different types of images, where previous work can typically only train on images of a similar appearance to those rendered by a $DR$.

Motivation & Objective

• Motivate reconstruction of 3D geometry from 2D images without ground-truth 3D models or lighting/texture supervision.
• Introduce a C∞ smooth differentiable renderer to provide gradients through occlusions.
• Close the training loop with image-to-image translators to bridge rendering domains.
• Develop a Reconstructive Adversarial Network (RAN) to train the reconstructor without 3D supervision.
• Demonstrate single- and multi-view reconstruction on synthetic and camera-captured data.

Proposed method

• Propose a C∞ smooth differentiable renderer (SR) based on soft blending of nearby triangles to ensure differentiability at occlusions.
• Train a pix2vex reconstructor to predict 3D vertex offsets from input images, using a base mesh and per-vertex updates.
• Use an image-to-image translator chain (a2b and b2a) to bridge domains between SR outputs and input images, enabling training with mismatched rendering styles.
• Employ a Reconstructive Adversarial Network (RAN) with multiple sub-RAN paths to self-supervise the translation components and the reconstructor.
• Leverage cycle-consistency and L1 losses across domains to stabilize training and avoid mode collapse.
• Process training with multi-view inputs where available, and propose strategies for single-view reconstruction.

Experimental results

Research questions

• RQ1 Can a smooth differentiable renderer provide usable gradients for 3D Reconstruction through occlusions?
• RQ2 Can a network predict 3D mesh geometry from images without ground-truth 3D supervision by using a RAN framework?
• RQ3 Does domain translation between renderer outputs and input images enable training with minimal supervision?
• RQ4 How well does the approach work with synthetic (ShapeNet) data and camera-captured images?
• RQ5 What are the effects of single-view versus multi-view inputs on reconstruction quality?

Key findings

• The proposed SR renderer is C∞ smooth, enabling differentiable gradients across occlusions.
• Pix2vex can predict 3D vertex offsets from images starting from a base mesh, yielding reconstructions without explicit 3D supervision.
• A two-image translator chain (a2b and b2a) bridges the SR output domain to the input domain, closing the training loop.
• The RAN framework enables unsupervised training of pix2vex and translators via adversarial objectives and cycle-consistency losses.
• Experiments show multi-view training on ShapeNet yields plausible reconstructions; single-view reconstructions from camera-captured shoes are demonstrated, highlighting robustness and limitations.