[Paper Review] Product Reservoir Computing: Time-Series Computation with Multiplicative Neurons

Alireza Goudarzi, Alireza Shabani+1 more|arXiv (Cornell University)|Feb 3, 2015

Neural Networks and Reservoir Computing27 references4 citations

TL;DR

This paper introduces Product Reservoir Computing (PRC), a novel reservoir computing architecture that replaces traditional tanh neurons with multiplicative (product) neurons to enhance nonlinear time-series computation. The method leverages the exponential-of-log trick for efficient dynamics simulation and demonstrates comparable performance to standard ESNs on chaotic time-series benchmarks like Mackey-Glass and Lorenz systems, while offering improved mathematical tractability and strong nonlinear capacity due to high-order statistical interactions.

ABSTRACT

Echo state networks (ESN), a type of reservoir computing (RC) architecture, are efficient and accurate artificial neural systems for time series processing and learning. An ESN consists of a core of recurrent neural networks, called a reservoir, with a small number of tunable parameters to generate a high-dimensional representation of an input, and a readout layer which is easily trained using regression to produce a desired output from the reservoir states. Certain computational tasks involve real-time calculation of high-order time correlations, which requires nonlinear transformation either in the reservoir or the readout layer. Traditional ESN employs a reservoir with sigmoid or tanh function neurons. In contrast, some types of biological neurons obey response curves that can be described as a product unit rather than a sum and threshold. Inspired by this class of neurons, we introduce a RC architecture with a reservoir of product nodes for time series computation. We find that the product RC shows many properties of standard ESN such as short-term memory and nonlinear capacity. On standard benchmarks for chaotic prediction tasks, the product RC maintains the performance of a standard nonlinear ESN while being more amenable to mathematical analysis. Our study provides evidence that such networks are powerful in highly nonlinear tasks owing to high-order statistics generated by the recurrent product node reservoir.

Motivation & Objective

• To explore whether multiplicative neurons—inspired by biological synaptic interactions—can serve as a viable alternative to standard activation functions in reservoir computing.
• To develop a computationally efficient and mathematically analyzable reservoir architecture using product nodes instead of sum-and-threshold neurons.
• To evaluate the performance of product-based reservoirs on standard nonlinear time-series prediction tasks, particularly chaotic systems.
• To investigate the stability and computational capacity of product reservoirs in comparison to traditional ESNs with tanh neurons.

Proposed method

• Replaces the standard tanh activation function in the reservoir with a product neuron model that computes a weighted product of inputs.
• Employs the exponential-of-log transformation to simulate product node dynamics efficiently using standard matrix operations.
• Adapts the echo state property proof to show that the product reservoir maintains stability and insensitivity to initial conditions under appropriate parameter settings.
• Uses a linear readout layer trained via regression to map reservoir states to desired outputs, preserving the core ESN training efficiency.
• Employs spectral radius and input weight scaling as key hyperparameters, with systematic tuning across benchmark tasks.
• Applies the method to standard benchmarks: Mackey-Glass and Lorenz systems, using normalized inputs and evaluating multi-step prediction via normalized mean squared error (NMSE).

Experimental results

Research questions

• RQ1 Can a reservoir composed of product neurons achieve performance comparable to standard ESNs with tanh neurons on nonlinear time-series prediction tasks?
• RQ2 How does the use of multiplicative neurons affect the stability and dynamical behavior of the reservoir, particularly in chaotic systems?
• RQ3 To what extent does the product reservoir leverage high-order statistical moments for improved nonlinear computation?
• RQ4 How do the hyperparameters (spectral radius and input weight) influence the performance and stability of the product reservoir versus standard ESNs?

Key findings

• The product reservoir achieves performance comparable to standard ESNs with tanh neurons on both the Mackey-Glass and Lorenz chaotic time-series prediction tasks.
• For both benchmarks, optimal performance was achieved at input weight ω = 0.1 and spectral radius λ = 0.8, indicating similar optimal parameter regimes.
• The product RC shows higher sensitivity to parameter changes than the tanh-ESN, particularly when λ is increased, where product RC dynamics become unstable, while tanh-ESN remains robust.
• On the Mackey-Glass task, decreasing λ increases error by 4 orders of magnitude in product RC, compared to 5 orders in tanh-ESN, indicating greater sensitivity in the product variant.
• For the Lorenz system, high ω and λ destabilize the product RC, leading to complete performance collapse, whereas the tanh-ESN remains stable and consistent.
• Multi-step prediction results show that product RC maintains similar NMSE performance to tanh-ESN across τ = 1 to 10 steps, confirming its capacity for long-term temporal computation.