[Paper Review] Follow the Gradient: Crossing the Reality Gap using Differentiable Physics (RealityGrad)

Jack Collins, Ross Brown+2 more|arXiv (Cornell University)|Sep 10, 2021

Robotic Path Planning Algorithms26 references4 citations

TL;DR

This paper introduces RealityGrad, a novel sim2real method that uses differentiable physics to iteratively optimize robot trajectories and refine simulation parameters via real-world rollouts. By combining gradient-based trajectory optimization with system identification, RealityGrad reduces task-space error by 66% in under 22 minutes per iteration on a desktop CPU, significantly closing the reality gap for robotic manipulators.

ABSTRACT

We propose a novel iterative approach for crossing the reality gap that utilises live robot rollouts and differentiable physics. Our method, RealityGrad, demonstrates for the first time, an efficient sim2real transfer in combination with a real2sim model optimisation for closing the reality gap. Differentiable physics has become an alluring alternative to classical rigid-body simulation due to the current culmination of automatic differentiation libraries, compute and non-linear optimisation libraries. Our method builds on this progress and employs differentiable physics for efficient trajectory optimisation. We demonstrate RealitGrad on a dynamic control task for a serial link robot manipulator and present results that show its efficiency and ability to quickly improve not just the robot's performance in real world tasks but also enhance the simulation model for future tasks. One iteration of RealityGrad takes less than 22 minutes on a desktop computer while reducing the error by 2/3, making it efficient compared to other sim2real methods in both compute and time. Our methodology and application of differentiable physics establishes a promising approach for crossing the reality gap and has great potential for scaling to complex environments.

Motivation & Objective

• Address the persistent reality gap in robotics, where policies trained in simulation fail when deployed on real robots due to discrepancies in dynamics and noise.
• Overcome limitations of existing sim2real methods like domain randomization, which often lead to conservative or inefficient policies.
• Develop an efficient, scalable, and iterative approach that improves both real-world policy performance and simulation model accuracy in tandem.
• Enable online, real-time adaptation of simulation models using real robot data and differentiable physics for gradient-based optimization.
• Demonstrate that differentiable simulators can be used not only for control optimization but also for accurate system identification to improve long-term sim2real transfer.

Proposed method

• Employ differentiable physics engines to compute gradients through simulation dynamics, enabling gradient-based optimization of control policies and model parameters.
• Use Model Predictive Control (MPC) with differentiable dynamics to generate K optimal trajectories in simulation using current model parameters.
• Train a neural network policy on the generated optimal trajectories to produce a differentiable policy network.
• Roll out the trained policy on a real robot to collect real-world trajectory data.
• Perform system identification in simulation by minimizing the difference between real robot trajectories and simulated trajectories using gradient descent on simulator parameters.
• Iterate the full pipeline—trajectory optimization, policy training, real rollout, and system identification—to progressively improve both policy performance and simulation fidelity.

Experimental results

Research questions

• RQ1 Can differentiable physics enable efficient, gradient-based optimization of robot trajectories and simulation parameters in a single, unified pipeline?
• RQ2 To what extent can real-world robot rollouts improve the accuracy of a differentiable physics simulator in modeling real robot dynamics?
• RQ3 How much can the reality gap be reduced in terms of task-space error using an iterative sim2real loop with differentiable physics?
• RQ4 Does the proposed method yield policies that are more efficient and less conservative than those trained with domain randomization?
• RQ5 Can the method be computationally efficient and run on standard desktop hardware without requiring high-end GPU acceleration?

Key findings

• RealityGrad reduces the accumulated Euclidean error between simulated and real robot trajectories from 129.79m (default parameters) to 47.81m after one iteration, achieving a 63% reduction in error.
• The residual error in system identification drops from 350,155.24 (default parameters) to 28.87 after one iteration, indicating strong alignment between simulation and real-world dynamics.
• After a second iteration, the residual error further decreases to 12.26, and the accumulated error between simulated and real trajectories drops to 59.98m, showing continued improvement.
• One full iteration of RealityGrad completes in less than 22 minutes on a desktop CPU (AMD 3970, 128GB RAM), demonstrating high computational efficiency.
• The method successfully identifies key physical parameters—especially joint damping values—showing that the simulation model can be accurately calibrated to real robot behavior.
• The improved simulation model enables better policy generalization and faster convergence, with policies showing more efficient and less oscillatory behavior in subsequent iterations.