[Paper Review] Universal Approximation of Input-Output Maps by Temporal Convolutional Nets

Joshua Hanson, Maxim Raginsky|arXiv (Cornell University)|Jun 21, 2019

Embedded Systems Design Techniques3 citations

TL;DR

This paper proves that Temporal Convolutional Networks (TCNs) can universally approximate a broad class of input-output maps with finite memory to arbitrary accuracy. It establishes quantitative approximation rates for deep ReLU TCNs in terms of network width, depth, and the modulus of continuity of the target map, extending their theoretical equivalence to recurrent networks for systems with limited long-term dependencies.

ABSTRACT

There has been a recent shift in sequence-to-sequence modeling from recurrent network architectures to convolutional network architectures due to computational advantages in training and operation while still achieving competitive performance. For systems having limited long-term temporal dependencies, the approximation capability of recurrent networks is essentially equivalent to that of temporal convolutional nets (TCNs). We prove that TCNs can approximate a large class of input-output maps having approximately finite memory to arbitrary error tolerance. Furthermore, we derive quantitative approximation rates for deep ReLU TCNs in terms of the width and depth of the network and modulus of continuity of the original input-output map, and apply these results to input-output maps of systems that admit finite-dimensional state-space realizations (i.e., recurrent models).

Motivation & Objective

• To establish the universal approximation capability of Temporal Convolutional Networks (TCNs) for input-output maps with approximately finite memory.
• To derive quantitative approximation error bounds for deep ReLU TCNs in terms of network width and depth.
• To demonstrate the theoretical equivalence between TCNs and recurrent networks in approximating systems with limited long-term temporal dependencies.
• To analyze approximation rates for maps arising from finite-dimensional state-space realizations, such as those in recurrent models.

Proposed method

• Theoretical analysis of TCNs using residual blocks with dilated causal convolutions to model long-range dependencies.
• Application of functional analysis techniques to prove universal approximation over a class of input-output maps with finite memory.
• Derivation of approximation error bounds based on the modulus of continuity of the target map and network depth/width.
• Use of ReLU activation functions to enable expressivity and tractable analysis of deep architectures.
• Formalization of the approximation problem in terms of function space embeddings and normed spaces.
• Comparison of TCN approximation performance with that of recurrent networks via theoretical equivalence under finite memory constraints.

Experimental results

Research questions

• RQ1 Can TCNs universally approximate a broad class of input-output maps with finite memory to arbitrary accuracy?
• RQ2 What are the quantitative approximation rates of deep ReLU TCNs in terms of network width and depth?
• RQ3 How do the approximation properties of TCNs compare to those of recurrent networks for systems with limited long-term dependencies?
• RQ4 To what extent can TCNs approximate maps derived from finite-dimensional state-space models?
• RQ5 How does the modulus of continuity of the input-output map influence the required network capacity for a given approximation tolerance?

Key findings

• TCNs can universally approximate any input-output map with approximately finite memory to arbitrary error tolerance.
• Approximation error decays as a function of network depth and width, with explicit dependence on the modulus of continuity of the target map.
• For maps with Hölder continuous regularity, the approximation rate improves with increasing depth and width, achieving polynomial convergence rates.
• The theoretical approximation capability of TCNs matches that of recurrent networks for systems with limited long-term temporal dependencies.
• The results provide a formal justification for the empirical success of TCNs in sequence-to-sequence modeling, especially in settings where long-range dependencies are weak.
• The framework applies directly to input-output maps arising from finite-dimensional state-space realizations, such as those in recurrent neural networks.