[Paper Review] Nonlinear Systems Identification Using Deep Dynamic Neural Networks

Olalekan Ogunmolu, Xuejun Gu+2 more|arXiv (Cornell University)|Oct 5, 2016

Neural Networks and Applications18 references106 citations

TL;DR

The paper investigates deep dynamic neural networks for modeling nonlinear dynamical systems, evaluating multilayer, recurrent, LSTM, GRU, and Hammerstein-style architectures on SISO/MIMO datasets (soft-robot and DaISy glass furnace) to assess identification performance and training efficiency.

ABSTRACT

Neural networks are known to be effective function approximators. Recently, deep neural networks have proven to be very effective in pattern recognition, classification tasks and human-level control to model highly nonlinear realworld systems. This paper investigates the effectiveness of deep neural networks in the modeling of dynamical systems with complex behavior. Three deep neural network structures are trained on sequential data, and we investigate the effectiveness of these networks in modeling associated characteristics of the underlying dynamical systems. We carry out similar evaluations on select publicly available system identification datasets. We demonstrate that deep neural networks are effective model estimators from input-output data

Motivation & Objective

• Motivate nonlinear system identification with deep networks as universal function approximators for sequential data.
• Evaluate multiple deep architectures (MLP, RNN, LSTM, GRU) for dynamic system modeling.
• Assess performance, training time, and robustness on SISO and MIMO datasets from soft-robot experiments and DaISy benchmarks.

Proposed method

• Use supervised learning to train deep networks to map inputs to outputs with mean-squared error loss.
• Compare feedforward MLP, simple RNN, LSTM variants, Fast LSTM, and GRU architectures on sequential data.
• Adopt Hammerstein-style models combining nonlinear static blocks with linear dynamic blocks.
• Train without batch normalization and with dropout in recurrent models to prevent overfitting.
• Provide architecture-specific equations for RNN/LSTM/GRU updates and Hammerstein structure.
• Report estimation fit and MSE, along with training times for each model and dataset.

Experimental results

Research questions

• RQ1 Can deep dynamic neural networks accurately identify nonlinear dynamical systems from input-output data across SISO and MIMO datasets?
• RQ2 Which neural architectures (MLP, RNN, LSTM variants, GRU, Hammerstein) balance fitting accuracy and training efficiency for dynamic system identification?
• RQ3 How do different architectures perform on real-world datasets (soft-robot head motion in radiotherapy and glass furnace temperature sensing) without preprocessing?
• RQ4 Do Hammerstein-style deep models offer advantages in capturing input-to-state delays and long-range mappings?

Key findings

• MLP and recurrent architectures achieve very high estimation fits on Soft-Robot and DaISy datasets, with MLP attaining the fastest training times among the tested models.
• Soft-Robot: MLP 99.8471% fit, 422s training, 113.1052 MSE; RNN 99.8448% fit, 1191s, 101.3624 MSE; LSTM 99.5144% fit, 876s, 382.6935 MSE; FastLSTM 99.4795% fit, 881s, 374.5386 MSE; GRU 99.835% fit, 884s, 117.0274 MSE.
• GlassFurnace (DaIsY): MLP 99.9905% fit, 462.64s, 0.015433 MSE; RNN 98.8052% fit, 726s, 0.245833 MSE; LSTM 57.1% fit, 4163.18s, 3.216 MSE; FastLSTM 91.2822% fit, 3310.56s, 0.689027 MSE.
• Hammerstein-style RNN/MLP models with 5-step backpropagation through time perform well and capture system dynamics efficiently compared to pure forward networks.
• LSTM-based Hammerstein and GRU-based Hammerstein variants provide strong modeling capability for temporal dependencies, with training time trade-offs.
• Results suggest deep dynamic NNs are effective, scalable alternatives to hand-crafted nonlinear system identification approaches.