[Paper Review] Architectural Complexity Measures of Recurrent Neural Networks

Saizheng Zhang, Yuhuai Wu+5 more|arXiv (Cornell University)|Feb 26, 2016

Adversarial Robustness in Machine Learning22 references111 citations

TL;DR

The paper introduces a graph-theoretic framework for RNN architectures and defines three complexity measures—recurrent depth, feedforward depth, and recurrent skip coefficient—and provides theoretical properties and empirical evidence that deeper recurrent and feedforward paths and higher skip coefficients can improve performance on sequential tasks.

ABSTRACT

In this paper, we systematically analyze the connecting architectures of recurrent neural networks (RNNs). Our main contribution is twofold: first, we present a rigorous graph-theoretic framework describing the connecting architectures of RNNs in general. Second, we propose three architecture complexity measures of RNNs: (a) the recurrent depth, which captures the RNN's over-time nonlinear complexity, (b) the feedforward depth, which captures the local input-output nonlinearity (similar to the "depth" in feedforward neural networks (FNNs)), and (c) the recurrent skip coefficient which captures how rapidly the information propagates over time. We rigorously prove each measure's existence and computability. Our experimental results show that RNNs might benefit from larger recurrent depth and feedforward depth. We further demonstrate that increasing recurrent skip coefficient offers performance boosts on long term dependency problems.

Motivation & Objective

• Provide a rigorous graph-based formulation of RNN connecting architectures.
• Define and prove the existence and computability of architectural complexity measures (recurrent depth, feedforward depth, recurrent skip coefficient).
• Empirically evaluate how varying depths and skip coefficients affect performance on sequential tasks.
• Illustrate that larger recurrent and feedforward depths can improve some tasks while excessive depth may hurt, and that higher skip coefficients aid long-term dependencies.

Proposed method

• Model the RNN connecting architecture as a weighted directed multigraph (RNN cyclic graph) and its time-unfolded DAG (RNN unfolded graph).
• Define recurrent depth d_r as the asymptotic average number of nonlinear transformations per time step via maximum cycle ratio l(θ)/σ_s(θ).
• Define feedforward depth d_f as the supremum of D*_i(n) − n·d_r over paths from inputs to outputs, computable as max over paths l(γ) − σ_s(γ)·d_r.
• Define recurrent skip coefficient s as the reciprocal of the minimal long-term path growth j = min_θ l(θ)/σ_s(θ) across cycles, i.e., s = 1/j.
• Prove existence and computability of d_r, d_f, and s under the general framework.
• Empirically compare architectures with varying d_r, d_f, and s on tasks such as Penn Treebank character-level modeling, text8, adding and copying memory problems, and sequential MNIST.

Experimental results

Research questions

• RQ1 Does recurrent depth capture meaningful over-time nonlinearity beyond traditional depth notions?
• RQ2 Do increases in recurrent depth and feedforward depth improve performance on sequential tasks, and are there optimal ranges?
• RQ3 Can increasing the recurrent skip coefficient improve learning of long-term dependencies?
• RQ4 How do these architectural measures relate to optimization difficulties in RNNs?

Key findings

• Recurrent depth d_r can differ across architectures with the same layer count; deeper recurrent structure leads to different long-term nonlinear growth.
• Increasing d_r and d_f can improve performance on some tasks (e.g., text8 with tanh RNNs and LSTMs), but too large d_r or d_f can hurt optimization and performance on others (e.g., sequential MNIST).
• Increasing the recurrent skip coefficient s substantially boosts performance on long-term dependency tasks such as the adding and copying memory problems and sequential MNIST, often outperforming baseline and comparable models.
• Skip connections that increase the effective recurrent skip coefficient (not all skip connections are equally beneficial) can yield large gains, highlighting the importance of considering s when designing skips.