[Paper Review] dynoNet: a neural network architecture for learning dynamical systems

Marco Forgione, Dario Piga|arXiv (Cornell University)|Jun 3, 2020

Neural Networks and Applications16 references4 citations

TL;DR

This paper proposes dynoNet, a differentiable neural network architecture that uses linear time-invariant (LTI) dynamical operators—parametrized as rational transfer functions (IIR filters)—as learnable layers for modeling sequential data and dynamical systems. By enabling end-to-end back-propagation through these IIR blocks, dynoNet achieves superior performance on system identification benchmarks with fewer parameters than FIR-based or RNN architectures, demonstrating state-of-the-art results on challenging electro-mechanical and nonlinear system datasets.

ABSTRACT

This paper introduces a network architecture, called dynoNet, utilizing linear dynamical operators as elementary building blocks. Owing to the dynamical nature of these blocks, dynoNet networks are tailored for sequence modeling and system identification purposes. The back-propagation behavior of the linear dynamical operator with respect to both its parameters and its input sequence is defined. This enables end-to-end training of structured networks containing linear dynamical operators and other differentiable units, exploiting existing deep learning software. Examples show the effectiveness of the proposed approach on well-known system identification benchmarks. Examples show the effectiveness of the proposed approach against well-known system identification benchmarks.

Motivation & Objective

• To develop a differentiable, end-to-end trainable neural network architecture tailored for sequence modeling and system identification.
• To address the limitations of standard 1D convolutional and recurrent networks in modeling long-term dependencies efficiently.
• To enable the integration of linear dynamical systems (IIR filters) as learnable components within deep learning frameworks via back-propagation.
• To demonstrate the representational power and generalization capability of the architecture on real-world system identification benchmarks.
• To provide a computationally efficient alternative to RNNs and FIR-based CNNs for modeling complex dynamical systems with fewer parameters.

Proposed method

• The core component is a differentiable linear dynamical operator defined by a rational transfer function, enabling infinite impulse response (IIR) filtering of input sequences.
• The forward and backward operations of the LTI layer are derived in closed-form to support efficient back-propagation using automatic differentiation.
• The architecture combines LTI blocks with static, differentiable components such as feed-forward networks, activation functions, and polynomials in a flexible, multi-input-multi-output (MIMO) structure.
• The network is trained end-to-end using standard deep learning software (e.g., PyTorch), leveraging reverse-mode automatic differentiation.
• The LTI layer is implemented as a recurrence relation based on linear difference equations, allowing vectorized processing of time series while preserving sequential dependencies.
• A software implementation is provided via a GitHub repository to support reproducibility and integration into existing DL pipelines.

Experimental results

Research questions

• RQ1 Can a differentiable IIR filter layer be effectively integrated into a deep learning framework for end-to-end training of dynamical system models?
• RQ2 How does the dynoNet architecture compare to standard 1D CNNs and RNNs in modeling long-term temporal dependencies with fewer parameters?
• RQ3 To what extent can dynoNet generalize to complex, nonlinear, and marginally stable dynamical systems such as electro-mechanical positioning systems?
• RQ4 Can the architecture outperform established black-box system identification models like NFIR, NARX, and NOBF on benchmark datasets?
• RQ5 How does the use of rational transfer functions in LTI blocks affect model accuracy and computational efficiency compared to FIR or recurrent architectures?

Key findings

• On the Wiener-Hammerstein circuit benchmark, dynoNet achieved a test fit of 93.2% and an RMSE of 4.52×10⁻⁵ mm, outperforming standard black-box models like NFIR (RMSE = 16.3×10⁻⁵ mm).
• For the Bouc-Wen hysteresis benchmark, the model achieved a test fit of 95.8% and an RMSE of 1.28×10⁻⁵ mm, demonstrating strong performance on nonlinear, memory-dependent systems.
• On the electro-mechanical positioning system (EMPS), dynoNet achieved a test fit of 96.8% and an RMSE of 2.64×10⁻³ mm, significantly outperforming the best linear model (fit = 25.4%).
• The architecture demonstrated superior sample efficiency and generalization on all benchmarks, particularly excelling in modeling systems with complex friction and integral dynamics.
• The computational cost of the LTI layer is lower than that of RNNs due to vectorized processing and reduced sequential dependency, while still capturing long-term dependencies via IIR filtering.
• The software implementation in PyTorch enables seamless integration with existing deep learning workflows and supports automatic differentiation for joint optimization of all network components.