[Paper Review] Nonlinear state-space identification using deep encoder networks

Gerben I. Beintema, Roland Tóth+1 more|arXiv (Cornell University)|Dec 14, 2020

Model Reduction and Neural Networks30 references4 citations

TL;DR

This paper proposes a deep encoder network to improve nonlinear state-space system identification by splitting data into sections for stochastic gradient optimization, reducing computational cost and smoothing the non-convex loss landscape. The encoder estimates initial states for each section using historical inputs and outputs, enabling efficient, scalable training and achieving the lowest known simulation error on the Wiener–Hammerstein benchmark.

ABSTRACT

Nonlinear state-space identification for dynamical systems is most often performed by minimizing the simulation error to reduce the effect of model errors. This optimization problem becomes computationally expensive for large datasets. Moreover, the problem is also strongly non-convex, often leading to sub-optimal parameter estimates. This paper introduces a method that approximates the simulation loss by splitting the data set into multiple independent sections similar to the multiple shooting method. This splitting operation allows for the use of stochastic gradient optimization methods which scale well with data set size and has a smoothing effect on the non-convex cost function. The main contribution of this paper is the introduction of an encoder function to estimate the initial state at the start of each section. The encoder function estimates the initial states using a feed-forward neural network starting from historical input and output samples. The efficiency and performance of the proposed state-space encoder method is illustrated on two well-known benchmarks where, for instance, the method achieves the lowest known simulation error on the Wiener--Hammerstein benchmark.

Motivation & Objective

• Address the high computational cost and non-convexity of simulation error minimization in nonlinear state-space system identification.
• Overcome the limitations of traditional methods that scale poorly with large datasets and suffer from local minima.
• Develop a scalable and stable optimization framework by combining multiple shooting with deep learning.
• Provide a generalizable method for initial state estimation across data sections without increasing model complexity.
• Achieve state-of-the-art performance on standard nonlinear system identification benchmarks.

Proposed method

• Split the full time series into N independent sections to enable stochastic gradient optimization.
• Use a feed-forward neural network (encoder) to estimate the initial state at the start of each section from historical input and output data.
• Formulate a multi-step-ahead prediction loss over each section, enabling end-to-end training of the model and encoder.
• Train the state-space model and encoder jointly using stochastic gradient descent, reducing computational cost per iteration.
• Apply the multiple shooting approach to smooth the loss function and improve convergence stability.
• Use the encoder to generalize to unseen data by enabling accurate state initialization without re-optimization.

Experimental results

Research questions

• RQ1 Can a deep encoder network effectively estimate initial states in nonlinear state-space models for improved optimization efficiency?
• RQ2 Does splitting the data into sections and using stochastic gradient descent reduce computational cost while maintaining accuracy?
• RQ3 How does the proposed method compare to existing approaches in terms of simulation error and convergence stability?
• RQ4 Can the encoder generalize to unseen data by enabling accurate state initialization?
• RQ5 What is the performance of the method on standard nonlinear system identification benchmarks like Wiener–Hammerstein and Silverbox?

Key findings

• The proposed method achieves the lowest known simulation error on the Wiener–Hammerstein benchmark, outperforming prior deep learning and system identification methods.
• The method significantly reduces computational cost by enabling stochastic gradient optimization over segmented data.
• The encoder-based initial state estimation avoids increasing model complexity with the number of sections.
• The multiple shooting approach with a learned encoder results in a smoother loss landscape, improving optimization stability.
• The method demonstrates competitive performance on the Silverbox benchmark, surpassing previously proposed deep learning methods.
• The encoder enables generalization to unseen datasets by accurately predicting initial states from input-output history.