[Paper Review] Scenario optimization for optimal training of Echo State Networks

Luca Bugliari Armenio, Lorenzo Fagiano+3 more|arXiv (Cornell University)|Dec 3, 2019

Neural Networks and Reservoir Computing20 references4 citations

TL;DR

This paper proposes a scenario optimization-based training method for Echo State Networks (ESNs) that replaces the traditional random parameter selection with a systematic, optimality-guaranteed approach. By treating the ESN training as a finite-sample optimization problem, the method ensures probabilistic bounds on performance metrics like RMSE and FIT, verified empirically on a pH neutralization process with 500 new scenarios showing violations below the 5% theoretical bound.

ABSTRACT

Echo State Networks (ESNs) are widely-used Recurrent Neural Networks. They are dynamical systems including, in state-space form, a nonlinear state equation and a linear output transformation. The common procedure to train ESNs is to randomly select the parameters of the state equation, and then to estimate those of the output equation via a standard least squares problem. Such a procedure is repeated for different instances of the random parameters characterizing the state equation, until satisfactory results are achieved. However, this trial-and-error procedure is not systematic and does not provide any guarantee about the optimality of the identification results. To solve this problem, we propose to complement the identification procedure of ESNs by applying results in scenario optimization. The resulting training procedure is theoretically sound and allows one to link precisely the number of identification instances to a guaranteed optimality bound on relevant performance indexes, such as the Root Mean Square error and the FIT index of the estimated model evaluated over a validation data-set. The proposed procedure is finally applied to the simulated model of a pH neutralization process: the obtained results confirm the validity of the approach.

Motivation & Objective

• To address the lack of optimality guarantees in standard Echo State Network (ESN) training, which relies on random initialization of state equation parameters.
• To develop a systematic, theoretically sound training procedure that ensures performance bounds on ESNs using scenario optimization theory.
• To link the number of training instances to precise probabilistic guarantees on key performance metrics like RMSE and FIT.
• To empirically validate the theoretical optimality bounds using a real-world SISO nonlinear benchmark: a pH neutralization process.

Proposed method

• Formulate the ESN training problem as a scalar optimization over an infinite set of possible random state equation parameterizations.
• Apply scenario optimization theory to transform the infinite-dimensional problem into a finite, convex optimization problem using Nδ sampled ESN instances.
• Select the best-performing ESN from Nδ instances based on validation data, with guaranteed upper bounds on the probability of a new ESN outperforming it.
• Use the least squares method to estimate the output layer weights for each sampled ESN, ensuring computational efficiency.
• Derive theoretical bounds on the violation probability ε for RMSE and FIT performance metrics based on the number of sampled instances Nδ.
• Validate the theoretical bounds by testing 500 new ESN instances and counting the number of violations of the derived performance thresholds.

Experimental results

Research questions

• RQ1 Can scenario optimization be effectively applied to ESN training to replace the heuristic random parameter selection with a theoretically guaranteed method?
• RQ2 What is the relationship between the number of sampled ESN instances (Nδ) and the resulting probabilistic bounds on RMSE and FIT performance metrics?
• RQ3 How conservatively do the theoretical violation bounds (ε) match empirical performance when testing new ESN instances?
• RQ4 Can the proposed method achieve reliable performance guarantees on a real-world nonlinear dynamical system, such as a pH neutralization process?

Key findings

• The proposed scenario optimization-based ESN training method achieves theoretical guarantees on performance, with the probability of a new ESN outperforming the selected one bounded by ε=5%.
• Empirical testing on 500 new ESN instances showed only 0.6% violations for the FIT metric and 0.8% for RMSE, well below the theoretical 5% bound.
• For ESNs with 200 states, the best FIT value was 89.15% and the worst RMSE was 0.048, with the theoretical bounds successfully capturing the performance limits.
• The method maintains computational efficiency by retaining the least squares estimation for output weights while adding rigorous optimality guarantees.
• The approach was successfully validated on a well-known SISO nonlinear benchmark (pH neutralization process), confirming the reliability of the derived performance bounds.
• The results demonstrate that the number of required training instances (Nδ=645) is sufficient to achieve tight, reliable performance guarantees with minimal empirical violations.