[Paper Review] Nonparametric Finite Time LTI System Identification

Tuhin Sarkar, Alexander Rakhlin+1 more|arXiv (Cornell University)|Feb 5, 2019

Control Systems and Identification31 references4 citations

TL;DR

This paper proposes a nonparametric finite-time system identification method for stable linear time-invariant (LTI) systems with unknown order, using a Hankel-like matrix constructed via ordinary least squares from noisy input-output data. It achieves accurate lower-order approximation by avoiding non-convex optimization, with theoretical guarantees on identification error within logarithmic factors of the statistical lower bound, and provides a data-dependent model order selection scheme closely tied to the Ho-Kalman algorithm.

ABSTRACT

We address the problem of learning the parameters of a stable linear time invariant (LTI) system or linear dynamical system (LDS) with unknown latent space dimension, or order, from a single time--series of noisy input-output data. We focus on learning the best lower order approximation allowed by finite data. Motivated by subspace algorithms in systems theory, where the doubly infinite system Hankel matrix captures both order and good lower order approximations, we construct a Hankel-like matrix from noisy finite data using ordinary least squares. This circumvents the non-convexities that arise in system identification, and allows accurate estimation of the underlying LTI system. Our results rely on careful analysis of self-normalized martingale difference terms that helps bound identification error up to logarithmic factors of the lower bound. We provide a data-dependent scheme for order selection and find an accurate realization of system parameters, corresponding to that order, by an approach that is closely related to the Ho-Kalman subspace algorithm. We demonstrate that the proposed model order selection procedure is not overly conservative, i.e., for the given data length it is not possible to estimate higher order models or find higher order approximations with reasonable accuracy.

Motivation & Objective

• To address finite-time system identification of stable LTI systems with unknown latent dimension (order) from single noisy input-output time series.
• To develop a nonparametric approach that adaptively selects the best model order based on finite data, avoiding reliance on prior knowledge of system order.
• To provide statistical guarantees on identification error for the best lower-order approximation, within logarithmic factors of the minimax lower bound.
• To construct a data-driven model order selection procedure that is not overly conservative, ensuring optimal use of available data length.

Proposed method

• Construct a Hankel-like matrix from finite noisy input-output data using ordinary least squares, circumventing non-convexities in system identification.
• Leverage the structure of the doubly infinite system Hankel matrix to capture both system order and lower-order approximations.
• Apply careful analysis of self-normalized martingale difference terms to bound the identification error up to logarithmic factors.
• Use a data-dependent scheme for model order selection based on singular values of the estimated Hankel matrix, inspired by the Ho-Kalman subspace algorithm.
• Formulate the system realization problem as a low-rank approximation task, enabling accurate estimation of system parameters (A, B, C) for the selected order.
• Employ active input design and apply Birge’s inequality and Le Cam’s method to derive information-theoretic lower bounds on identification error.

Experimental results

Research questions

• RQ1 Can a nonparametric system identification method achieve accurate lower-order approximation of a high-order LTI system from finite noisy data without prior knowledge of the system order?
• RQ2 What is the fundamental statistical limit of identification error for finite-time LTI system identification, and can the proposed method achieve this up to logarithmic factors?
• RQ3 How can a data-driven model order selection procedure be designed to avoid overfitting and ensure that no higher-order model can be estimated with reasonable accuracy for the given data length?
• RQ4 To what extent can the Ho-Kalman subspace algorithm be adapted for finite-data, noisy system identification with provable statistical guarantees?
• RQ5 Is it possible to construct a non-convex system identification framework that avoids local minima by using a least-squares Hankel matrix estimator?

Key findings

• The proposed method achieves identification error within logarithmic factors of the minimax lower bound, establishing statistical optimality up to logarithmic terms.
• The data-dependent model order selection procedure is not overly conservative, meaning that for the given data length T, no higher-order model can be estimated with reasonable accuracy.
• The use of ordinary least squares on a Hankel-like matrix effectively bypasses non-convex optimization, enabling globally optimal estimation without local minima issues.
• Theoretical analysis confirms that the identification error is bounded by a term scaling as O((log T)/T) under appropriate assumptions, with the bound depending on the condition number of the Hankel matrix.
• The method successfully identifies a low-rank realization of the system that approximates the true system with high fidelity, even when the true order is high and unknown.
• The construction of canonical systems with Hankel matrices of known rank and condition number enables tight lower bounds on identification error, validating the theoretical claims.