[Paper Review] Differentiable programming and its applications to dynamical systems

Adrián Hernández, José M. Amigó|arXiv (Cornell University)|Dec 17, 2019

Reinforcement Learning in Robotics49 references6 citations

TL;DR

This paper introduces differentiable programming as an extension of deep learning that integrates algorithmic components—such as attention mechanisms, memory networks, and differentiable ODE solvers—into end-to-end differentiable models. By leveraging automatic differentiation, the framework enhances traditional RNNs and LSTMs in modeling, predicting, and interpreting dynamical systems, particularly by enabling reasoning, selective attention to relevant time steps, and efficient learning with prior physical structure.

ABSTRACT

Differentiable programming is the combination of classical neural networks modules with algorithmic ones in an end-to-end differentiable model. These new models, that use automatic differentiation to calculate gradients, have new learning capabilities (reasoning, attention and memory). In this tutorial, aimed at researchers in nonlinear systems with prior knowledge of deep learning, we present this new programming paradigm, describe some of its new features such as attention mechanisms, and highlight the benefits they bring. Then, we analyse the uses and limitations of traditional deep learning models in the modeling and prediction of dynamical systems. Here, a dynamical system is meant to be a set of state variables that evolve in time under general internal and external interactions. Finally, we review the advantages and applications of differentiable programming to dynamical systems.

Motivation & Objective

• To address the limitations of traditional deep learning in modeling complex dynamical systems, particularly in handling long sequences and time-varying dependencies.
• To introduce differentiable programming as a paradigm that extends deep learning with reasoning, memory, and attention capabilities.
• To demonstrate how differentiable components—like memory networks and ODE solvers—can be integrated into deep learning models for improved performance in dynamical systems.
• To provide a foundation for researchers in nonlinear systems to apply differentiable programming tools to scientific modeling and simulation.

Proposed method

• Combines classical neural networks with differentiable algorithmic modules (e.g., attention, memory, ODE solvers) into a single end-to-end differentiable computational graph.
• Employs automatic differentiation (via frameworks like PyTorch) to compute gradients through complex, hybrid models including neural controllers and physical simulators.
• Integrates differentiable ODE solvers (e.g., Euler method) into deep learning models, where the ODE's right-hand side is a neural network, enabling gradient-based optimization of system parameters.
• Uses differentiable physical models such as DiffTaichi and differentiable physics engines to embed known physical laws into learning architectures.
• Applies memory networks and Differentiable Neural Computers (DNCs) to store and selectively retrieve historical data from dynamical systems.
• Employs self-attention mechanisms to allow models to dynamically focus on relevant time steps or inputs during prediction.

Experimental results

Research questions

• RQ1 How can differentiable programming improve the modeling and prediction of dynamical systems compared to standard RNNs and LSTMs?
• RQ2 In what ways do attention mechanisms and memory networks enhance the learning and interpretability of dynamical systems in deep learning models?
• RQ3 Can differentiable ODE solvers and physical models be effectively integrated into end-to-end training to improve sample efficiency and generalization?
• RQ4 What are the theoretical and practical limitations of current differentiable programming models when applied to dynamical systems?

Key findings

• Differentiable programming enables end-to-end training of hybrid models that combine neural networks with algorithmic components such as attention and memory, leading to improved reasoning and selective feature focus.
• Attention mechanisms allow models to dynamically identify and prioritize relevant time steps or inputs, improving performance on long sequences where standard RNNs degrade.
• Differentiable memory networks and DNCs enhance long-term dependency modeling by storing and retrieving historical system states, improving prediction accuracy in complex dynamical systems.
• Integrating differentiable ODE solvers into deep learning models allows for efficient optimization of physical parameters (e.g., trebuchet design) by backpropagating through the simulator, reducing reliance on brute-force search.
• Differentiable physical models, such as DiffTaichi and differentiable physics engines, enable learning of physical parameters while preserving the underlying physical structure, improving generalization and interpretability.
• The framework shows promise in scientific computing, enabling the creation of data-parametrized, differentiable architectures that incorporate known physical laws, thus accelerating simulation and learning.