[Paper Review] Regularized system identification using orthonormal basis functions

Tianshi Chen, Lennart Ljung|arXiv (Cornell University)|Apr 11, 2015

Control Systems and Identification9 references3 citations

TL;DR

This paper proposes a regularized system identification framework using orthonormal basis functions, extending ridge regularization from impulse response estimation to general basis function expansions. It shows that kernel-based regularization via orthonormal bases is equivalent to ridge regression on coefficients, with superior performance over traditional FIR and Laguerre models, especially for slow systems with limited data.

ABSTRACT

Most of existing results on regularized system identification focus on regularized impulse response estimation. Since the impulse response model is a special case of orthonormal basis functions, it is interesting to consider if it is possible to tackle the regularized system identification using more compact orthonormal basis functions. In this paper, we explore two possibilities. First, we construct reproducing kernel Hilbert space of impulse responses by orthonormal basis functions and then use the induced reproducing kernel for the regularized impulse response estimation. Second, we extend the regularization method from impulse response estimation to the more general orthonormal basis functions estimation. For both cases, the poles of the basis functions are treated as hyperparameters and estimated by empirical Bayes method. Then we further show that the former is a special case of the latter, and more specifically, the former is equivalent to ridge regression of the coefficients of the orthonormal basis functions.

Motivation & Objective

• Address the limitations of traditional system identification methods in short and noisy data scenarios.
• Overcome the high variance of finite impulse response (FIR) models for systems with slow dynamics or high sampling rates.
• Develop a unified regularization framework for orthonormal basis functions that improves model accuracy and robustness.
• Investigate the role of basis function poles and kernel structure in regularization performance.
• Demonstrate that regularization on basis function coefficients outperforms impulse response regularization in certain cases.

Proposed method

• Construct a reproducing kernel Hilbert space (RKHS) for impulse responses using orthonormal basis functions to enable kernel-based regularization.
• Extend regularization from impulse response estimation to general orthonormal basis function expansions, treating basis poles as hyper-parameters.
• Use empirical Bayes to estimate the poles of the orthonormal basis functions, optimizing them for data fit.
• Formulate the regularization as a kernel ridge regression problem on the coefficients of the basis functions.
• Compare two kernels: the Tikhonov-regularized (TC) kernel and the diagonal (DI) kernel, for their impact on coefficient estimation.
• Apply the framework to Laguerre basis functions as a case study, evaluating performance against FIR and standard Laguerre models.

Experimental results

Research questions

• RQ1 Can regularization via orthonormal basis functions improve system identification accuracy compared to standard FIR models?
• RQ2 Is the kernel-based regularization on impulse responses equivalent to regularization on the coefficients of orthonormal basis functions?
• RQ3 How do different kernel structures (TC vs. DI) affect the performance of regularized basis function estimation?
• RQ4 Does regularizing the coefficients of orthonormal basis functions outperform traditional impulse response regularization?
• RQ5 How does the choice of basis function poles impact the final model performance when estimated via empirical Bayes?

Key findings

• The proposed method, RLAG (regularized Laguerre basis functions), achieves model fits of 91.8% (S1D1), 76.3% (S1D2), 86.8% (S2D1), and 70.1% (S2D2), significantly outperforming LS-LAG and RFIR in slow system cases.
• For slow systems (S2D1 and S2D2), RLAG improves performance by approximately 5% over RFIR, demonstrating the benefit of compact basis representation.
• The TC and DI kernels yield nearly identical performance in RLAG, suggesting that independence assumptions on coefficients are reasonable when absolute convergence is ensured.
• RFIR-LAG performs worse than RLAG, indicating that ridge regression is an ill-suited regularization for Laguerre coefficients due to mismatched structure.
• The RKHS-based regularization using orthonormal bases is mathematically equivalent to ridge regression on basis coefficients, validating the theoretical foundation.
• Empirical Bayes estimation of basis function poles leads to improved model fit, confirming the importance of adaptive hyper-parameter tuning.