[Paper Review] On the Computational Power of RNNs

Samuel A. Korsky, Robert C. Berwick|arXiv (Cornell University)|Jun 14, 2019

Machine Learning and Algorithms13 references19 citations

TL;DR

This paper establishes the computational limits of practical recurrent neural networks (RNNs) and Gated Recurrent Units (GRUs). It proves that finite-precision RNNs with ReLU activation are as powerful as deterministic finite automata (DFAs), while infinite-precision RNNs with ReLU can recognize context-free languages (equivalent to pushdown automata). GRUs with infinite precision and nonlinear outputs also achieve pushdown automaton power, demonstrating that standard architectures can, in theory, capture hierarchical linguistic structures.

ABSTRACT

Recent neural network architectures such as the basic recurrent neural network (RNN) and Gated Recurrent Unit (GRU) have gained prominence as end-to-end learning architectures for natural language processing tasks. But what is the computational power of such systems? We prove that finite precision RNNs with one hidden layer and ReLU activation and finite precision GRUs are exactly as computationally powerful as deterministic finite automata. Allowing arbitrary precision, we prove that RNNs with one hidden layer and ReLU activation are at least as computationally powerful as pushdown automata. If we also allow infinite precision, infinite edge weights, and nonlinear output activation functions, we prove that GRUs are at least as computationally powerful as pushdown automata. All results are shown constructively.

Motivation & Objective

• To determine the theoretical computational power of standard RNNs and GRUs used in practice, particularly in NLP.
• To investigate whether these models can recognize context-free languages, such as those with nested or balanced structures.
• To clarify the role of activation functions, precision, and architecture in determining computational expressivity.
• To bridge the gap between empirical success in NLP and theoretical understanding of what these models can compute.

Proposed method

• Constructing a finite automaton from the hidden states of a finite-precision RNN to prove equivalence with deterministic finite automata (DFAs).
• Using a constructive proof to show that infinite-precision RNNs with ReLU activation can simulate pushdown automata (PDAs).
• Demonstrating that GRUs with infinite precision, infinite edge weights, and nonlinear output functions can also simulate PDAs.
• Analyzing the evolution of hidden states across inputs to show how RNNs and GRUs maintain and update state information for language recognition.
• Defining acceptance via final output values in a predetermined set S, enabling formal language recognition.
• Applying theoretical models from automata theory to analyze the computational capacity of neural network architectures.

Experimental results

Research questions

• RQ1 Can finite-precision RNNs with ReLU activation recognize all regular languages?
• RQ2 Is there a theoretical limit to the computational power of standard RNNs when precision is constrained?
• RQ3 Can infinite-precision RNNs with ReLU activation recognize context-free languages such as Dyck languages?
• RQ4 Do GRUs with standard activation functions and infinite precision achieve the expressivity of pushdown automata?
• RQ5 What architectural or precision conditions are necessary for RNNs or GRUs to simulate PDAs?

Key findings

• Finite-precision RNNs with ReLU activation and one hidden layer are exactly as computationally powerful as deterministic finite automata (DFAs).
• Infinite-precision RNNs with ReLU activation are at least as powerful as pushdown automata (PDAs), meaning they can recognize context-free languages.
• GRUs with infinite precision, infinite edge weights, and nonlinear output functions are also at least as powerful as PDAs.
• The proof is constructive, showing explicit network configurations that simulate PDAs for languages like a^n b^n.
• The results hold under standard assumptions of sequential input processing and common activation functions, unlike prior works that required special activation functions.
• The study clarifies that standard RNNs and GRUs, when given sufficient precision, can in principle model hierarchical linguistic structures.