[Paper Review] Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer

Wenzheng Chen, Jun Gao+5 more|arXiv (Cornell University)|Aug 3, 2019

Computer Graphics and Visualization TechniquesComputer Science38 references220 citations

TL;DR

DIB-R introduces an analytic, interpolation-based differentiable rasterizer that enables gradients through foreground vertex attribute interpolation and distance-based background aggregation, enabling 2D-supervised learning of 3D geometry, texture, and lighting.

ABSTRACT

Many machine learning models operate on images, but ignore the fact that images are 2D projections formed by 3D geometry interacting with light, in a process called rendering. Enabling ML models to understand image formation might be key for generalization. However, due to an essential rasterization step involving discrete assignment operations, rendering pipelines are non-differentiable and thus largely inaccessible to gradient-based ML techniques. In this paper, we present {\emph DIB-R}, a differentiable rendering framework which allows gradients to be analytically computed for all pixels in an image. Key to our approach is to view foreground rasterization as a weighted interpolation of local properties and background rasterization as a distance-based aggregation of global geometry. Our approach allows for accurate optimization over vertex positions, colors, normals, light directions and texture coordinates through a variety of lighting models. We showcase our approach in two ML applications: single-image 3D object prediction, and 3D textured object generation, both trained using exclusively using 2D supervision. Our project website is: https://nv-tlabs.github.io/DIB-R/

Motivation & Objective

• Motivate learning 3D geometry from 2D images by making rendering differentiable.
• Develop an interpolation-based rasterizer that provides analytic gradients for all pixels.
• Enable optimization over vertex positions, colors, normals, texture coordinates, and lighting.
• Demonstrate single-image 3D object prediction and 3D textured object generation using only 2D supervision.

Proposed method

• Define foreground rasterization as a weighted barycentric interpolation of vertex attributes for the closest enclosing face.
• Introduce background rasterization as a distance-based aggregation over global geometry with a soft alpha assignment (A_i').
• Compute analytic gradients for all vertex attributes (positions, colors, textures, normals, lighting) via chain rule through the interpolated pixel values.
• Support multiple lighting models (Phong, Lambertian, Spherical Harmonics) by separating mesh color and lighting factors in the fragment stage.
• Train networks to predict geometry, texture maps, and lighting from 2D inputs, using 2D supervision and multi-view consistency, plus adversarial and perceptual losses for realism.
• Extend to a 3D GAN setup to generate textured 3D shapes using only 2D supervision.

Experimental results

Research questions

• RQ1 Can a differentiable, interpolation-based rasterizer provide complete analytical gradients for all relevant vertex attributes in a rasterized scene?
• RQ2 How well can 3D geometry, texture, and lighting be inferred from single 2D images using a differentiable renderer?
• RQ3 Does 2D supervision suffice to train models for predicting 3D shapes and textures, and can adversarial training further improve realism?
• RQ4 Can the approach be extended to generative modeling of textured 3D objects using 2D supervision?

Key findings

• DIB-R achieves state-of-the-art results among rasterization-based renderers on single-image 3D object prediction in 13 ShapeNet categories, outperforming N3MR and SoftRas-Mesh.
• Predictions with geometry and color (and later texture and light) show quantitative improvements in 3D IOU and F-score across most categories.
• Texture and lighting predictions outperform baselines on 3D shape, texture, and lighting accuracy, with notable gains in texture L1 differences and lighting direction errors.
• Incorporating adversarial losses improves texture realism and helps disentangle texture from lighting in rendered outputs.
• The method supports 3D object generation (3D GAN) under 2D supervision, producing diverse, high-quality textured shapes.