[Paper Review] High-order Differentiable Autoencoder for Nonlinear Model Reduction

Siyuan Shen, Yin Yang+5 more|arXiv (Cornell University)|Feb 19, 2021

Model Reduction and Neural Networks4 citations

TL;DR

This paper proposes a high-order differentiable deep autoencoder (DAE) for nonlinear model reduction in physics-based simulation, enabling real-time elastic deformation by efficiently computing second- and third-order derivatives via complex-step finite difference (CSFD) combined with reverse automatic differentiation. The method achieves real-time performance with deep, high-dimensional DAEs (10+ layers) on GPU, outperforming linear reduction in capturing complex nonlinear dynamics.

ABSTRACT

This paper provides a new avenue for exploiting deep neural networks to improve physics-based simulation. Specifically, we integrate the classic Lagrangian mechanics with a deep autoencoder to accelerate elastic simulation of deformable solids. Due to the inertia effect, the dynamic equilibrium cannot be established without evaluating the second-order derivatives of the deep autoencoder network. This is beyond the capability of off-the-shelf automatic differentiation packages and algorithms, which mainly focus on the gradient evaluation. Solving the nonlinear force equilibrium is even more challenging if the standard Newton's method is to be used. This is because we need to compute a third-order derivative of the network to obtain the variational Hessian. We attack those difficulties by exploiting complex-step finite difference, coupled with reverse automatic differentiation. This strategy allows us to enjoy the convenience and accuracy of complex-step finite difference and in the meantime, to deploy complex-value perturbations as collectively as possible to save excessive network passes. With a GPU-based implementation, we are able to wield deep autoencoders (e.g., $10+$ layers) with a relatively high-dimension latent space in real-time. Along this pipeline, we also design a sampling network and a weighting network to enable \emph{weight-varying} Cubature integration in order to incorporate nonlinearity in the model reduction. We believe this work will inspire and benefit future research efforts in nonlinearly reduced physical simulation problems.

Motivation & Objective

• To address the limitation of linear model reduction in capturing complex, nonlinear dynamics of deformable solids.
• To enable accurate, real-time physics-based simulation using deep autoencoders by overcoming the computational challenge of high-order derivative evaluation.
• To develop a differentiable DAE framework that supports second- and third-order derivatives required for Newtonian dynamics in subspace simulation.
• To integrate neural Cubature and domain decomposition for efficient, localized nonlinear reduction in complex geometries.
• To demonstrate the feasibility and performance of high-order differentiable DAEs in real-time animation and simulation.

Proposed method

• Augments deep autoencoders with complex-step finite difference (CSFD) to compute high-order derivatives efficiently, enabling accurate projection into time-varying tangent spaces.
• Combines CSFD with reverse automatic differentiation to evaluate first-, second-, and third-order derivatives of the DAE with only O(n) network passes.
• Designs a neural Cubature network with learnable weights to enable adaptive, non-uniform sampling and accurate integration in the reduced space.
• Employs domain decomposition with deformation substructuring to apply local nonlinear subspaces, improving efficiency and realism in complex models.
• Uses generalized Newton-Euler equations to couple local deformation with global rigid body motion for free-floating simulations.
• Deploys a GPU-accelerated pipeline to achieve real-time simulation with deep, high-dimensional DAEs (e.g., 10+ layers, latent space > 50).

Figure 1. In this paper, we exploit deep autoencoder (DAE) networks to accelerate physics-based simulation. In order to model nonlinear subspace dynamics accurately, second- and high-order derivatives of the deep decoder net must be efficiently evaluated to match the subspace simulation frame rate.

Experimental results

Research questions

• RQ1 Can high-order derivatives of deep autoencoders be efficiently computed to enable accurate nonlinear model reduction in physics-based simulation?
• RQ2 How can complex-step finite difference be adapted to deep neural networks to support second- and third-order derivative evaluation for Newtonian dynamics?
• RQ3 Can a differentiable DAE be integrated into Lagrangian mechanics to simulate elastic deformations in real time?
• RQ4 How does the proposed method compare to linear model reduction in capturing complex, nonlinear deformation behaviors?
• RQ5 Can domain decomposition and neural Cubature enhance the scalability and accuracy of nonlinear model reduction in large, geometrically complex models?

Key findings

• The proposed CSFD-augmented DAE enables real-time simulation of nonlinear deformable solids with deep autoencoders (10+ layers) and high-dimensional latent spaces.
• The method achieves real-time performance (FPS) on complex models such as the puffer ball (320 strings) and falling bunny with 30+ DOFs, outperforming single-core full simulation.
• The use of neural Cubature with learnable weights improves integration accuracy and sampling efficiency in nonlinear reduced spaces.
• Domain decomposition with local nonlinear subspaces (e.g., n_p=10, n_q=6 per domain) enables scalable and detailed simulation of complex models like cacti.
• The framework supports robust simulation of free-floating objects using generalized Newton-Euler equations, with collision handling via implicit penalty forces.
• The method demonstrates that high-order differentiability via CSFD enables accurate modeling of inertia effects in nonlinear subspace dynamics, which standard AD tools cannot support.

Figure 2. Increased nonlinearity better fits training poses but also makes the network more wiggling.