[Paper Review] gradSim: Differentiable simulation for system identification and visuomotor control

J. Krishna Murthy, Miles Macklin+12 more|arXiv (Cornell University)|Apr 6, 2021

Advanced Vision and Imaging109 references43 citations

TL;DR

∇Sim is a differentiable simulator that backpropagates from video pixels through differentiable physics and rendering to estimate physical properties and enable visuomotor control without 3D supervision.

ABSTRACT

We consider the problem of estimating an object's physical properties such as mass, friction, and elasticity directly from video sequences. Such a system identification problem is fundamentally ill-posed due to the loss of information during image formation. Current solutions require precise 3D labels which are labor-intensive to gather, and infeasible to create for many systems such as deformable solids or cloth. We present gradSim, a framework that overcomes the dependence on 3D supervision by leveraging differentiable multiphysics simulation and differentiable rendering to jointly model the evolution of scene dynamics and image formation. This novel combination enables backpropagation from pixels in a video sequence through to the underlying physical attributes that generated them. Moreover, our unified computation graph -- spanning from the dynamics and through the rendering process -- enables learning in challenging visuomotor control tasks, without relying on state-based (3D) supervision, while obtaining performance competitive to or better than techniques that rely on precise 3D labels.

Motivation & Objective

• Motivate and address the ill-posed problem of estimating object physical properties from videos without 3D supervision.
• Couple differentiable multiphysics simulation with differentiable rendering to form an end-to-end learnable pipeline.
• Enable recovering friction, elasticity, deformable material parameters, and visuomotor controls from image-space supervision.
• Demonstrate competitive performance against methods requiring 3D labels on rigid, deformable, and cloth models.

Proposed method

• Model a unified simulation pipeline that maps simulation state and parameters to rendered images via differentiable physics and rendering components.
• Use implicit relationships from step-based dynamics and a differentiable rasterization approach to backpropagate through both dynamics and rendering.
• Employ a differentiable physics engine with state s=(q,u) and parameters θ, and render via SoftRas or DIB-R to obtain differentiable image outputs.
• Train end-to-end with losses defined in image space, enabling backpropagation from pixels to physical attributes.
• Support meshes and tetrahedral FEM for rigid bodies, deformable solids, and thin-shell cloth, with adjustable render rates for efficiency.

Experimental results

Research questions

• RQ1 Can physical parameters be accurately identified from video by backpropagating through the differentiable simulator?
• RQ2 What is the performance gap between image-space (2D) supervision and state-space (3D) supervision in differentiable physics-based estimation?
• RQ3 Are learning signals from gradients through the unified simulator effective for challenging visuomotor control tasks?
• RQ4 How do modeling choices in dynamics and rendering affect parameter identification and control performance?

Key findings

• ∇Sim can recover mass, friction, and elasticity from video with high precision, competitive with 3D-supervised methods.
• For deformable and cloth models, per-particle mass and Lamé parameters can be estimated from video, with favorable relative errors compared to baselines.
• The loss landscape under ∇Sim is smooth, enabling effective optimization with gradient-based methods, unlike non-differentiable baselines.
• In visuomotor control tasks, networks guided by ∇Sim gradients achieve target-oriented control using only image-based supervision, sometimes rivaling 3D-supervised approaches.
• Imperfect dynamics modeling degrades parameter recovery more than rendering imperfections, and shading/texture cues accelerate convergence.