[Paper Review] Embedding Theory of Reservoir Computing and Reducing Reservoir Network Using Time Delays

Xing-Yue Duan, Xiong Ying+4 more|arXiv (Cornell University)|Mar 16, 2023

Neural Networks and Reservoir Computing4 citations

TL;DR

This paper establishes a rigorous embedding theory framework for reservoir computing (RC), proving RC inherently performs high-dimensional embedding of input dynamical systems. By introducing time delays only in the output layer, it enables a trade-off between time delay and reservoir size, reducing network complexity while enhancing memory capacity—demonstrated by single-neuron RC outperforming large standard RCs in reconstruction and prediction tasks.

ABSTRACT

Reservoir computing (RC), a particular form of recurrent neural network, is under explosive development due to its exceptional efficacy and high performance in reconstruction or/and prediction of complex physical systems. However, the mechanism triggering such effective applications of RC is still unclear, awaiting deep and systematic exploration. Here, combining the delayed embedding theory with the generalized embedding theory, we rigorously prove that RC is essentially a high dimensional embedding of the original input nonlinear dynamical system. Thus, using this embedding property, we unify into a universal framework the standard RC and the time-delayed RC where we novelly introduce time delays only into the network's output layer, and we further find a trade-off relation between the time delays and the number of neurons in RC. Based on this finding, we significantly reduce the network size of RC for reconstructing and predicting some representative physical systems, and, more surprisingly, only using a single neuron reservoir with time delays is sometimes sufficient for achieving those tasks.

Motivation & Objective

• To uncover the fundamental mechanism behind reservoir computing's efficacy in reconstructing and predicting complex dynamical systems.
• To unify standard and time-delayed RC frameworks under a single theoretical embedding framework.
• To reduce reservoir network size without sacrificing performance by exploiting time delays.
• To establish a trade-off between time delay and neuron count for optimal memory capacity.
• To validate the framework on high-dimensional spatiotemporal systems like ISCAM and chaotic attractors.

Proposed method

• Combines delayed embedding theory and generalized embedding theory to prove that RC is a high-dimensional embedding of the input dynamical system.
• Introduces a novel time-delayed RC architecture where time delays are applied only to the output layer, not the reservoir itself.
• Derives a theoretical trade-off relation between time delay and reservoir neuron count for maintaining system reconstruction performance.
• Uses dimension tests and delayed mutual information (DMI) to select optimal reservoir dimension (d) and time delay (τ).
• Applies constraints based on Lyapunov time to ensure predictive power of delayed observables (τ·Δt·N_lag < Λ_max).
• Employs least-squares training for the output layer, preserving computational efficiency despite reduced reservoir size.

Experimental results

Research questions

• RQ1 How does reservoir computing mathematically relate to nonlinear dynamical system embedding?
• RQ2 Can time delays in the output layer replace large reservoirs in achieving system reconstruction and prediction?
• RQ3 What is the theoretical trade-off between time delay and reservoir size in RC?
• RQ4 Can a single-neuron reservoir with time delays outperform large standard RCs?
• RQ5 How can hyper-parameters (d, τ) be optimally selected for effective system reconstruction?

Key findings

• Reservoir computing is rigorously proven to be a high-dimensional embedding of the input dynamical system, providing a theoretical foundation for its efficacy.
• A single-neuron reservoir with time delays can achieve reconstruction and prediction tasks that require large standard reservoirs without delays.
• The framework enables a significant reduction in reservoir size—e.g., 1000 neurons with 5 lags outperformed a 5000-neuron standard RC in reconstructing the 100×100 ISCAM model.
• Time-delayed RC maintains nearly identical reconstruction accuracy to standard RC with the same output dimension, even on high-dimensional chaotic systems.
• The trade-off between time delay and neuron count allows for memory capacity preservation while drastically reducing computational and hardware costs.
• Optimal hyper-parameters (d and τ) can be selected using dimension tests and modified DMI, with τ constrained by the system’s Lyapunov time to preserve predictive power.

Figure 4: Reconstructed chaotic patterns for the ISCAM model by a standard non-delayed RC including 5000 neurons and a time-delayed RC including 1000 neurons and 5 lags for each neuron. (a) Selected dynamical pattern using different evolution rules. (b) Reconstruction errors deviating from true dyna