[Paper Review] RISP: Rendering-Invariant State Predictor with Differentiable Simulation and Rendering for Cross-Domain Parameter Estimation

Pingchuan Ma, Tao Du+3 more|arXiv (Cornell University)|May 11, 2022

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes RISP, a rendering-invariant state predictor that leverages differentiable simulation and rendering gradients to estimate physical system parameters from videos under unknown rendering conditions. By training a network to be invariant to lighting, materials, and shadows using a novel loss based on rendering gradients, RISP achieves state-of-the-art performance in cross-domain state estimation, system identification, imitation learning, and visuomotor control with significantly lower reconstruction errors than prior methods.

ABSTRACT

This work considers identifying parameters characterizing a physical system's dynamic motion directly from a video whose rendering configurations are inaccessible. Existing solutions require massive training data or lack generalizability to unknown rendering configurations. We propose a novel approach that marries domain randomization and differentiable rendering gradients to address this problem. Our core idea is to train a rendering-invariant state-prediction (RISP) network that transforms image differences into state differences independent of rendering configurations, e.g., lighting, shadows, or material reflectance. To train this predictor, we formulate a new loss on rendering variances using gradients from differentiable rendering. Moreover, we present an efficient, second-order method to compute the gradients of this loss, allowing it to be integrated seamlessly into modern deep learning frameworks. We evaluate our method in rigid-body and deformable-body simulation environments using four tasks: state estimation, system identification, imitation learning, and visuomotor control. We further demonstrate the efficacy of our approach on a real-world example: inferring the state and action sequences of a quadrotor from a video of its motion sequences. Compared with existing methods, our approach achieves significantly lower reconstruction errors and has better generalizability among unknown rendering configurations.

Motivation & Objective

• To address the challenge of estimating physical system parameters from video when rendering configurations (e.g., lighting, materials) are unknown and uncontrolled.
• To overcome the limitations of existing methods that require known renderers or large-scale domain adaptation data.
• To develop a differentiable, end-to-end framework that generalizes across diverse rendering domains without requiring access to target-domain data.
• To enable robust parameter estimation in real-world scenarios where rendering and physical dynamics are complex and unmodeled.
• To integrate rendering gradients into a state prediction network to enforce invariance to rendering variations.

Proposed method

• Propose a rendering-invariant state predictor (RISP) that maps image differences to state differences independent of rendering configurations.
• Formulate a novel loss function based on gradients of the rendering parameters (e.g., lighting, material) to enforce invariance in the state prediction network.
• Use a second-order optimization method to efficiently compute gradients of the rendering-variance loss, enabling integration into deep learning frameworks.
• Train RISP using domain randomization, generating diverse rendering conditions (lighting, backgrounds, materials) during training to cover the distribution of unknown test configurations.
• Leverage differentiable simulation and rendering to backpropagate through the entire pipeline, enabling end-to-end optimization of the state predictor.
• Integrate the RISP network into downstream tasks such as system identification, imitation learning, and visuomotor control using differentiable dynamics.

Experimental results

Research questions

• RQ1 Can a neural network be trained to predict physical system states from video in a way that is invariant to unknown rendering configurations such as lighting and material reflectance?
• RQ2 How can rendering gradients be effectively used to enforce invariance in a state prediction network without requiring access to target-domain data?
• RQ3 Does incorporating rendering gradients into the loss function improve generalization across unseen rendering domains compared to pixel-wise or perceptual losses?
• RQ4 To what extent can RISP generalize to real-world videos with complex textures and unmodeled dynamics, such as a flying quadrotor?
• RQ5 How does the proposed method compare in performance to baselines that rely on domain adaptation or known rendering settings?

Key findings

• RISP achieves significantly lower reconstruction errors than pixelwise-loss and perceptual-loss baselines across all tested environments and rendering configurations.
• In imitation learning, RISP reduces state discrepancy by over 80% compared to the next-best baseline, with an average error of 1.52 ± 0.18 in the hand environment.
• The ablation study confirms that rendering gradients are essential to performance, as removing them increases error by over 10x in some tasks.
• On the real-world quadrotor task, RISP successfully reconstructs action sequences that closely match the motion in the reference video, outperforming all baselines despite unmodeled aerodynamic effects and complex textures.
• The method generalizes well to unseen rendering domains, demonstrating robustness even when training and test configurations differ significantly.
• The use of second-order gradients enables efficient and stable training of the rendering-variance loss, making it feasible to integrate into deep learning pipelines.