[Paper Review] On the Practical Computational Power of Finite Precision RNNs for Language Recognition

Gail Garfinkel Weiss, Yoav Goldberg+1 more|arXiv (Cornell University)|May 13, 2018

Neural Networks and Applications6 citations

TL;DR

This paper investigates the practical computational power of finite-precision RNNs in language recognition, showing that LSTMs and ReLU-based Elman-RNNs can implement unbounded counting due to their gating mechanisms, while GRUs and squashing-activation RNNs cannot. The key contribution is demonstrating that LSTMs can recognize context-sensitive languages like $a^n b^n c^n$ under input-bound, finite-precision constraints, whereas GRUs remain finite-state and fail on such tasks.

ABSTRACT

While Recurrent Neural Networks (RNNs) are famously known to be Turing complete, this relies on infinite precision in the states and unbounded computation time. We consider the case of RNNs with finite precision whose computation time is linear in the input length. Under these limitations, we show that different RNN variants have different computational power. In particular, we show that the LSTM and the Elman-RNN with ReLU activation are strictly stronger than the RNN with a squashing activation and the GRU. This is achieved because LSTMs and ReLU-RNNs can easily implement counting behavior. We show empirically that the LSTM does indeed learn to effectively use the counting mechanism.

Motivation & Objective

• To analyze the computational power of finite-precision RNNs under realistic NLP training constraints, such as linear computation time and 32-bit floating-point precision.
• To determine whether different RNN variants—LSTM, GRU, ReLU-RNN, and squashing-RNN—differ in their ability to recognize context-sensitive languages like $a^n b^n c^n$.
• To investigate whether backpropagation-trained RNNs can learn and utilize explicit counting mechanisms in practice, especially under finite-precision constraints.
• To formally prove that SRNNs with squashing activations cannot implement a bidirectional binary counter, even with auxiliary dimensions, while LSTMs and ReLU-RNNs can.
• To provide empirical evidence that LSTMs learn and use dedicated counting dimensions in their hidden states, unlike GRUs.

Proposed method

• The study focuses on Input-Bound Finite-Precision RNNs (IBFP-RNNs), where computation time is linear in input length and states are represented with 32-bit floating-point precision.
• Theoretical analysis proves that Elman-RNNs with squashing activations (e.g., tanh) cannot implement a bidirectional binary counter due to sign-preserving constraints in the tanh function and linear transformations.
• The proof uses contradiction: assuming a counter that increases on input 'up' and decreases on 'down' leads to conflicting sign requirements on the bias difference $b_{up} - b_{down}$, violating linearity.
• Empirical evaluation trains LSTMs and GRUs on $a^n b^n$ and $a^n b^n c^n$ languages, visualizing hidden state activations to show that LSTMs use specific dimensions for counting.
• The analysis extends to ReLU-RNNs, showing they can emulate k-counter machines and thus outperform squashing-RNNs in computational power.
• Theoretical and empirical comparisons are made between LSTM, GRU, ReLU-RNN, and squashing-RNN, focusing on their ability to simulate unbounded counting under finite precision.

Experimental results

Research questions

• RQ1 Can finite-precision, input-bound RNNs with backpropagation training recognize context-sensitive languages like $a^n b^n c^n$?
• RQ2 Do LSTMs and ReLU-RNNs have greater computational power than GRUs and squashing-RNNs under finite-precision constraints?
• RQ3 Can LSTMs learn and utilize explicit counting mechanisms in their hidden states during training, as evidenced by activation patterns?
• RQ4 Is it theoretically impossible for Elman-RNNs with squashing activations to implement a bidirectional binary counter, even with auxiliary dimensions?
• RQ5 Does the presence of gating mechanisms in LSTMs enable unbounded counting, while GRUs remain finite-state under the same constraints?

Key findings

• LSTMs and ReLU-based Elman-RNNs can implement unbounded counting under finite-precision and input-bound constraints, enabling recognition of context-sensitive languages like $a^n b^n c^n$.
• GRUs and squashing-activation Elman-RNNs cannot implement a bidirectional binary counter due to sign-preserving constraints in the tanh activation and linear transformations, making them finite-state.
• Empirical results show that LSTMs learn to dedicate specific hidden state dimensions to counting, as evidenced by distinct activation patterns on $a^n b^n$ and $a^n b^n c^n$ sequences.
• Theoretical analysis proves that no Elman-RNN with squashing activation can implement a consistent bidirectional counter, even with additional dimensions, due to contradictory sign requirements in the bias difference.
• ReLU-RNNs, like LSTMs, can emulate k-counter machines and thus have greater computational power than squashing-RNNs, despite not having explicit gates.
• The findings suggest that LSTMs are uniquely suited for tasks requiring counting, such as processing linearized parse trees, which may explain their empirical success in such NLP applications.