[Paper Review] DRWR: A Differentiable Renderer without Rendering for Unsupervised 3D Structure Learning from Silhouette Images

Zhizhong Han, Chao Chen+2 more|arXiv (Cornell University)|Jul 12, 2020

Advanced Vision and ImagingComputer Science55 references22 citations

TL;DR

DRWR proposes a differentiable renderer without rendering for unsupervised 3D point cloud reconstruction from silhouette images, using a smooth silhouette loss to pull 3D point projections into the foreground and a structure-aware repulsion loss to prevent point clumping. It achieves state-of-the-art accuracy with significantly faster training by eliminating pixel-wise interpolation, visibility handling, and shading.

ABSTRACT

Differentiable renderers have been used successfully for unsupervised 3D structure learning from 2D images because they can bridge the gap between 3D and 2D. To optimize 3D shape parameters, current renderers rely on pixel-wise losses between rendered images of 3D reconstructions and ground truth images from corresponding viewpoints. Hence they require interpolation of the recovered 3D structure at each pixel, visibility handling, and optionally evaluating a shading model. In contrast, here we propose a Differentiable Renderer Without Rendering (DRWR) that omits these steps. DRWR only relies on a simple but effective loss that evaluates how well the projections of reconstructed 3D point clouds cover the ground truth object silhouette. Specifically, DRWR employs a smooth silhouette loss to pull the projection of each individual 3D point inside the object silhouette, and a structure-aware repulsion loss to push each pair of projections that fall inside the silhouette far away from each other. Although we omit surface interpolation, visibility handling, and shading, our results demonstrate that DRWR achieves state-of-the-art accuracies under widely used benchmarks, outperforming previous methods both qualitatively and quantitatively. In addition, our training times are significantly lower due to the simplicity of DRWR.

Motivation & Objective

• To address the high computational cost of rendering in differentiable renderers for 3D structure learning.
• To investigate whether pixel-wise interpolation, visibility handling, and shading are essential for accurate 3D reconstruction from silhouettes.
• To develop a lightweight, differentiable renderer that bypasses rendering while maintaining or improving reconstruction quality.
• To enable efficient and robust training for 3D point cloud reconstruction using only 2D silhouette supervision.

Proposed method

• DRWR replaces traditional pixel-wise losses with a projection-based loss that operates directly on 2D projections of 3D points.
• It employs a smooth silhouette loss that provides non-zero gradients until 3D point projections enter the foreground, mitigating local minima.
• A structure-aware repulsion loss adaptively pushes pairs of projected points apart only when both lie inside the silhouette, improving coverage and distribution.
• The method avoids surface interpolation, visibility handling (e.g., z-buffering), and shading, eliminating rendering overhead.
• The loss function combines unary and pairwise components: the unary loss pulls points into the silhouette, and the pairwise loss enforces spatial spread within the foreground.
• Optimization is performed end-to-end using backpropagation, with the differentiable nature of the loss enabling gradient-based learning of 3D point cloud parameters.

Experimental results

Research questions

• RQ1 Can 3D structure learning be achieved without rendering by relying solely on 2D projections of 3D points?
• RQ2 Does eliminating pixel-wise interpolation, visibility handling, and shading degrade or improve reconstruction accuracy?
• RQ3 Can a smooth silhouette loss effectively resolve local minima issues in silhouette-based 3D reconstruction?
• RQ4 How does structure-aware repulsion between projected points improve shape coverage and generalization?
• RQ5 What is the trade-off between training efficiency and reconstruction accuracy when removing the rendering pipeline?

Key findings

• DRWR achieves state-of-the-art performance on standard benchmarks, outperforming prior methods both quantitatively and qualitatively in 3D point cloud reconstruction from silhouettes.
• The ablation study shows that removing the smooth silhouette loss or the structure-aware repulsion leads to significant performance drops, confirming their necessity.
• Training with DRWR is up to 2× faster than DPC (a prior state-of-the-art point-based differentiable renderer), with training times reduced from ~72 hours to ~36 hours for 16,000-point reconstructions at 128² resolution.
• DRWR generalizes well to real images, producing high-fidelity reconstructions from Internet-sourced silhouettes without fine-tuning.
• Latent space interpolation demonstrates that DRWR learns a meaningful, continuous latent space, enabling smooth shape interpolation between learned shapes.
• The loss function with structure awareness reduces conflict between the smooth silhouette and repulsion losses, leading to faster convergence and lower training loss.