[Paper Review] Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning

Shichen Liu, Tianye Li+2 more|arXiv (Cornell University)|Apr 3, 2019

Advanced Vision and Imaging47 references129 citations

TL;DR

Soft Rasterizer presents a fully differentiable rendering framework that treats rendering as a soft, probabilistic aggregation across all mesh triangles, enabling end-to-end gradient flow from pixels to 3D mesh properties for unsupervised single-view reconstruction and image-based shape fitting.

ABSTRACT

Rendering bridges the gap between 2D vision and 3D scenes by simulating the physical process of image formation. By inverting such renderer, one can think of a learning approach to infer 3D information from 2D images. However, standard graphics renderers involve a fundamental discretization step called rasterization, which prevents the rendering process to be differentiable, hence able to be learned. Unlike the state-of-the-art differentiable renderers, which only approximate the rendering gradient in the back propagation, we propose a truly differentiable rendering framework that is able to (1) directly render colorized mesh using differentiable functions and (2) back-propagate efficient supervision signals to mesh vertices and their attributes from various forms of image representations, including silhouette, shading and color images. The key to our framework is a novel formulation that views rendering as an aggregation function that fuses the probabilistic contributions of all mesh triangles with respect to the rendered pixels. Such formulation enables our framework to flow gradients to the occluded and far-range vertices, which cannot be achieved by the previous state-of-the-arts. We show that by using the proposed renderer, one can achieve significant improvement in 3D unsupervised single-view reconstruction both qualitatively and quantitatively. Experiments also demonstrate that our approach is able to handle the challenging tasks in image-based shape fitting, which remain nontrivial to existing differentiable renderers.

Motivation & Objective

• Motivate 3D reasoning from 2D images by enabling dense, pixel-to-3D supervision via a differentiable renderer.
• Address non-differentiability of rasterization by reformulating rendering as a soft probabilistic aggregation.
• Enable gradient flow to occluded and far-range mesh vertices for robust 3D reconstruction and fitting.
• Support rendering of colorized meshes, silhouettes, and shading to supervise mesh geometry and appearance.

Proposed method

• Introduce probability maps D_j per triangle j to model each pixel’s likelihood of being influenced by that triangle.
• Define a differentiable aggregate function A that fuses per-triangle color maps using D_j and triangle depths (z_j) to produce the final image.
• Use a sigmoid-based soft rasterization with a controllable sharpness parameter sigma and a small gamma to balance smoothness and fidelity.
• Provide variants of the aggregate function (A_O for silhouettes, A_S for color) and option to use a neural-network-based aggregator A_N for comparison.
• Train with rendering losses on silhouettes and/or color images, plus optional geometry (Laplacian) regularization to stabilize reconstruction.
• Demonstrate applicability to single-view mesh reconstruction and image-based shape fitting with occlusions and non-rigid deformations.

Experimental results

Research questions

• RQ1 Can a truly differentiable rendering layer enable end-to-end training for 3D mesh reconstruction from a single image without 3D supervision?
• RQ2 How can occlusions and far-range vertices receive gradient signals through a soft, probabilistic rasterization framework?
• RQ3 Does soft rasterization provide smoother optimization landscapes and improved robustness for image-based shape fitting compared to prior differentiable renderers?
• RQ4 What is the impact of different distance metrics and aggregation functions on 3D reconstruction quality?
• RQ5 How well does the method texture reconstructed meshes using a color palette approach from input images?

Key findings

• SoftRas achieves superior mean IoU on ShapeNet categories compared with unsupervised baselines, approaching or surpassing some supervised methods in single-view reconstruction.
• The probabilistic aggregation enables gradients to flow to occluded and far-range triangles, improving pose fitting and non-rigid shape optimization.
• Color reconstruction via a learned color palette yields sharp texture on reconstructed meshes from low-resolution input images.
• Ablation shows Euclidean distance and the A_O/A_S aggregators perform well, with a neural-network-based A_N offering marginal gains at higher computation cost.
• Rendering with silhouette and color losses yields better performance than silhouette-only supervision, and Laplacian geometry regularization provides extra gains.