[Paper Review] Differentiable reservoir computing

Lyudmila Grigoryeva, Juan‐Pablo Ortega|Alexandria (UniSG) (University of St.Gallen)|Feb 16, 2019

Neural Networks and Reservoir Computing66 references4 citations

TL;DR

This paper establishes the differentiability of reservoir filters in discrete-time reservoir computing systems under general conditions, deriving a Volterra-type series representation for analytic reservoir maps. The key contribution is a rigorous Taylor-series-based approximation with explicit error bounds, proving that any fading memory filter can be uniformly approximated by a finite Volterra series with finite memory.

ABSTRACT

Much effort has been devoted in the last two decades to characterize the situations in which a reservoir computing system exhibits the so-called echo state (ESP) and fading memory (FMP) properties. These important features amount, in mathematical terms, to the existence and continuity of global reservoir system solutions. That research is complemented in this paper with the characterization of the differentiability of reservoir filters for very general classes of discrete-time deterministic inputs. This constitutes a novel strong contribution to the long line of research on the ESP and the FMP and, in particular, links to existing research on the input-dependence of the ESP. Differentiability has been shown in the literature to be a key feature in the learning of attractors of chaotic dynamical systems. A Volterra-type series representation for reservoir filters with semi-infinite discrete-time inputs is constructed in the analytic case using Taylor's theorem and corresponding approximation bounds are provided. Finally, it is shown as a corollary of these results that any fading memory filter can be uniformly approximated by a finite Volterra series with finite memory.

Motivation & Objective

• To characterize the differentiability of reservoir filters for general discrete-time deterministic inputs.
• To extend existing research on the echo state property (ESP) and fading memory property (FMP) by analyzing input-dependence and smoothness.
• To provide a Volterra series representation for reservoir filters under analyticity assumptions, enabling systematic approximation.
• To establish that any fading memory filter can be uniformly approximated by a finite-memory Volterra series.
• To bridge theoretical reservoir computing with practical learning by ensuring differentiability, crucial for training chaotic dynamical systems.

Proposed method

• Uses Taylor's theorem to derive a Volterra-type series representation for reservoir filters in the analytic case.
• Defines the reservoir filter as a solution map from input sequences to reservoir states, ensuring differentiability under analyticity and norm constraints.
• Applies Cauchy bounds for analytic functions to derive explicit error estimates for truncated Volterra expansions.
• Introduces weighted sequence spaces $\ell_w^{-}({\mathbb{R}})$ to control input and state norms, ensuring convergence.
• Leverages the echo state property (ESP) to guarantee unique, globally defined reservoir filter solutions.
• Proves time-invariance of reservoir system solutions via shift operators $T_\tau$, preserving system dynamics under time shifts.

Experimental results

Research questions

• RQ1 Under what conditions is the reservoir filter differentiable with respect to input sequences?
• RQ2 Can a Volterra series representation be rigorously constructed for reservoir filters under analyticity assumptions?
• RQ3 What are the approximation error bounds when truncating the Volterra series to finite memory?
• RQ4 How does differentiability of the filter relate to the fading memory property (FMP) and echo state property (ESP)?
• RQ5 Is it possible to uniformly approximate any fading memory filter using a finite Volterra series with finite memory?

Key findings

• A Volterra-type series representation for reservoir filters is constructed using Taylor's theorem, valid for analytic reservoir maps.
• Explicit error bounds are derived using Cauchy estimates, showing that the approximation error decays geometrically with truncation order.
• The error bound is proportional to $\frac{L}{w_{-t}} \left(1 - \frac{\|\mathbf{z}\|_w}{M}\right)^{-1} \left(\frac{\|\mathbf{z}\|_w}{M}\right)^{p+1}$, where $L$ and $M$ are analyticity constants.
• The differentiability of the reservoir filter is established under analyticity and norm-boundedness assumptions on the reservoir map.
• Any fading memory filter can be uniformly approximated by a finite Volterra series with finite memory, as a corollary of the main results.
• The time-invariance of reservoir system solutions is proven without requiring the echo state property, relying only on the system dynamics.