[论文解读] Deep networks for system identification: a Survey

Gianluigi Pillonetto, Aleksandr Y. Aravkin+4 more|arXiv (Cornell University)|Jan 30, 2023

Neural Networks and Applications被引用 5

一句话总结

本综述全面概述了用于系统辨识的深度学习技术，弥合了经典方法与现代神经网络之间的差距。结果表明，诸如前馈网络、循环网络和状态空间网络等深度架构即使在过参数化的情况下，也能有效从输入-输出数据中建模动态系统，这得益于隐式正则化以及良性过拟合等泛化现象。

ABSTRACT

Deep learning is a topic of considerable current interest. The availability of massive data collections and powerful software resources has led to an impressive amount of results in many application areas that reveal essential but hidden properties of the observations. System identification learns mathematical descriptions of dynamic systems from input-output data and can thus benefit from the advances of deep neural networks to enrich the possible range of models to choose from. For this reason, we provide a survey of deep learning from a system identification perspective. We cover a wide spectrum of topics to enable researchers to understand the methods, providing rigorous practical and theoretical insights into the benefits and challenges of using them. The main aim of the identified model is to predict new data from previous observations. This can be achieved with different deep learning based modelling techniques and we discuss architectures commonly adopted in the literature, like feedforward, convolutional, and recurrent networks. Their parameters have to be estimated from past data trying to optimize the prediction performance. For this purpose, we discuss a specific set of first-order optimization tools that is emerged as efficient. The survey then draws connections to the well-studied area of kernel-based methods. They control the data fit by regularization terms that penalize models not in line with prior assumptions. We illustrate how to cast them in deep architectures to obtain deep kernel-based methods. The success of deep learning also resulted in surprising empirical observations, like the counter-intuitive behaviour of models with many parameters. We discuss the role of overparameterized models, including their connection to kernels, as well as implicit regularization mechanisms which affect generalization, specifically the interesting phenomena of benign overfitting ...

研究动机与目标

• 通过分析深度神经网络如何从输入-输出数据中建模动态系统，弥合深度学习与系统辨识之间的鸿沟。
• 探讨在系统辨识中使用过参数化深度网络的理论与实际挑战，包括泛化与优化问题。
• 将深度学习技术与经典正则化及核方法相联系，展示深度核模型如何扩展传统方法。
• 识别将现代架构（如Transformer、归一化流、生成对抗网络和图神经网络）应用于非线性系统辨识的开放问题。
• 研究过参数化在动态系统建模中的理论基础，如双 descent 和良性过拟合。

提出的方法

• 调研在系统辨识中常用深度神经网络架构：前馈网络、卷积网络、循环网络和状态空间网络。
• 采用一阶优化方法（例如随机梯度下降）通过最小化观测数据上的预测误差来训练深度模型。
• 通过将深度架构与施加系统行为先验假设的正则化项结合，构建深度核基模型。
• 使用网格搜索与交叉验证来调整超参数，并在训练集与验证集上评估模型性能。
• 将经典系统辨识原理（如预测误差最小化与模型阶次选择）整合进深度学习框架。
• 通过在40%的保留验证集上进行经验评估来展示泛化能力，使用决定系数等标准指标衡量拟合程度。

实验结果

研究问题

• RQ1 在黑箱系统辨识框架下，深度神经网络如何被有效适配以建模动态系统？
• RQ2 过参数化与隐式正则化在深度学习模型于系统辨识中泛化过程中的作用是什么？
• RQ3 深度核基方法如何扩展系统辨识中的经典正则化技术？
• RQ4 如Transformer、归一化流和图神经网络等现代深度学习架构能否成功应用于非线性系统辨识？
• RQ5 哪些理论基础可解释过参数化深度模型在动态系统建模中的泛化行为？

主要发现

• 即使仅用300个样本进行训练，具有超过10万个参数的深度前馈网络、LSTM和状态空间模型在验证数据上仍表现出良好泛化能力，在未见数据上取得了高拟合分数。
• 在验证集上，深度模型的预测精度优于三阶连续线性基线模型，证明了深度架构在数据有限情况下的强大能力。
• 基于25次随机权重初始化的箱线图结果显示各次运行性能一致，表明深度学习方法具有鲁棒性与稳定性。
• 实证结果支持关于良性过拟合与双 descent 的理论见解，即参数量超过数据点数的模型仍能良好泛化。
• 正则化与深度学习框架的结合，可在不降低模型维度的情况下有效控制模型复杂度。
• 如Transformer、归一化流、生成对抗网络和图神经网络等现代架构在未来的系统辨识任务中展现出巨大潜力，尽管其应用仍处于探索阶段。