[Paper Review] Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction

Shichen Liu, Weikai Chen+2 more|arXiv (Cornell University)|Jan 17, 2019

3D Shape Modeling and Analysis57 references78 citations

TL;DR

Introduces Soft Rasterizer (SoftRas), a differentiable silhouette-based rasterizer that enables unsupervised single-view 3D mesh reconstruction by backpropagating rendering loss to a mesh generator.

ABSTRACT

Rendering is the process of generating 2D images from 3D assets, simulated in a virtual environment, typically with a graphics pipeline. By inverting such renderer, one can think of a learning approach to predict a 3D shape from an input image. However, standard rendering pipelines involve a fundamental discretization step called rasterization, which prevents the rendering process to be differentiable, hence suitable for learning. We present the first non-parametric and truly differentiable rasterizer based on silhouettes. Our method enables unsupervised learning for high-quality 3D mesh reconstruction from a single image. We call our framework `soft rasterizer' as it provides an accurate soft approximation of the standard rasterizer. The key idea is to fuse the probabilistic contributions of all mesh triangles with respect to the rendered pixels. When combined with a mesh generator in a deep neural network, our soft rasterizer is able to generate an approximated silhouette of the generated polygon mesh in the forward pass. The rendering loss is back-propagated to supervise the mesh generation without the need of 3D training data. Experimental results demonstrate that our approach significantly outperforms the state-of-the-art unsupervised techniques, both quantitatively and qualitatively. We also show that our soft rasterizer can achieve comparable results to the cutting-edge supervised learning method and in various cases even better ones, especially for real-world data.

Motivation & Objective

• Motivate unsupervised 3D reconstruction from a single image when 3D ground truth is unavailable.
• Propose a truly differentiable rasterizer suitable for end-to-end learning.
• Show that silhouette-based supervision with SoftRas yields high-quality mesh reconstructions.
• Compare against state-of-the-art unsupervised methods and analyze realism and geometry quality.

Proposed method

• Formulate rasterization as a probabilistic, differentiable process via per-triangle distance-based probability maps D_j over image pixels.
• Define D_j^i = sigmoid(delta_ij * d(i,j)^2 / sigma) where d(i,j) is the distance from pixel i to triangle f_j and delta_ij indicates whether the pixel is inside or outside f_j.
• Aggregate triangle contributions with a differentiable OR-like operator hat{S}^i = 1 - prod_j (1 - D_j^i) to approximate silhouette formation.
• Integrate SoftRas into an end-to-end mesh reconstruction network that deforms a template sphere using an encoder-decoder to predict per-vertex displacements.
• Train using IoU-based silhouette loss between rendered hat{S} and ground-truth silhouette S, augmented with Laplacian and flattening regularizers for smooth geometry.
• Optionally extend to color reconstruction by backpropagating color losses through the differentiable rasterizer.

Experimental results

Research questions

• RQ1 Can a truly differentiable rasterizer approximate standard rasterization well enough to supervise 3D mesh generation from silhouettes alone?
• RQ2 How does silhouette-based supervision with SoftRas compare to existing unsupervised and supervised 3D reconstruction methods in terms of geometry quality and IoU?
• RQ3 What is the impact of geometry regularizers (Laplacian, flattening) and multi-view training on reconstruction accuracy?
• RQ4 How well does the approach generalize from synthetic silhouettes to real images?

Key findings

• SoftRas achieves competitive IoU with state-of-the-art unsupervised methods and can approach, or in some cases surpass, supervised results on several categories.
• The probabilistic, differentiable rasterization yields smooth, high-quality meshes with fewer artifacts (e.g., self-intersections) compared to prior differentiable renderers.
• Using more training views improves reconstruction accuracy and can mitigate silhouette-based ambiguities on objects with large planar regions.
• Ablation shows both Laplacian and flattening losses are important for producing smooth, non-intersecting geometries.
• Color can be recovered by backpropagating color loss through the SoftRas layer, enabling colorized mesh reconstruction.
• Table-based results indicate SoftRas attains mean IoU improvements over several baselines across ShapeNet categories.