[Paper Review] An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling

Shaojie Bai, J. Zico Kolter+1 more|arXiv (Cornell University)|Mar 4, 2018

Topic Modeling66 references4,314 citations

TL;DR

This paper compares a simple, generic Temporal Convolutional Network (TCN) against LSTM/GRU/RNN baselines across multiple sequence modeling tasks, finding TCNs often outperform recurrent models and exhibit longer practical memory.

ABSTRACT

For most deep learning practitioners, sequence modeling is synonymous with recurrent networks. Yet recent results indicate that convolutional architectures can outperform recurrent networks on tasks such as audio synthesis and machine translation. Given a new sequence modeling task or dataset, which architecture should one use? We conduct a systematic evaluation of generic convolutional and recurrent architectures for sequence modeling. The models are evaluated across a broad range of standard tasks that are commonly used to benchmark recurrent networks. Our results indicate that a simple convolutional architecture outperforms canonical recurrent networks such as LSTMs across a diverse range of tasks and datasets, while demonstrating longer effective memory. We conclude that the common association between sequence modeling and recurrent networks should be reconsidered, and convolutional networks should be regarded as a natural starting point for sequence modeling tasks. To assist related work, we have made code available at http://github.com/locuslab/TCN .

Motivation & Objective

• Motivate re-evaluating the default use of recurrent networks for sequence modeling.
• Provide a simple, canonical TCN architecture as a starting point for sequence tasks.
• Systematically compare TCNs with canonical RNNs across diverse benchmarks (music, language, synthetic tests).
• Analyze memory retention differences between TCNs and RNNs to understand long-range dependencies.

Proposed method

• Propose a generic Temporal Convolutional Network (TCN) with causal 1D convolutions and zero-padding to maintain sequence length.
• Incorporate dilated causal convolutions to achieve very long receptive fields while preserving causality.
• Use residual blocks with 1x1 projections to match layer dimensions and enable deep architectures.
• Apply weight normalization and spatial dropout for regularization within TCNs.
• Compare TCNs to LSTM, GRU, and vanilla RNN baselines using identical model sizes and standard optimization (Adam, gradient clipping).
• Evaluate on a broad set of tasks including polyphonic music modeling, word- and character-level language modeling, and synthetic stress tests.

Experimental results

Research questions

• RQ1 Can a generic TCN with causal, dilated convolutions outperform canonical recurrent architectures on standard sequence modeling benchmarks?
• RQ2 Do convolutional networks retain information over longer effective histories than LSTM/GRU/RNN models in practice?
• RQ3 How do TCNs perform on tasks traditionally used to benchmark recurrent networks (music, language modeling, synthetic memory tasks)?
• RQ4 What are the practical trade-offs of TCNs versus recurrent models in terms of memory usage and training stability?

Key findings

• TCNs substantially outperform canonical recurrent networks across a broad range of sequence modeling tasks.
• In synthetic stress tests like the adding problem and copy memory, TCNs converge faster and achieve lower losses than LSTMs/GRUs.
• On Sequential MNIST and permuted MNIST, TCNs achieve higher accuracy and faster convergence.
• On polyphonic music (JSB Chorales, Nottingham) and word/character-level language modeling (PTB, WikiText-103, LAMBADA, text8), TCNs match or exceed recurrent baselines without task-specific tuning.
• TCNs demonstrate longer effective memory than RNNs of comparable size, evidenced by copy memory and LAMBADA results.
• Overall, the paper argues that convolutional architectures should be a natural starting point for sequence modeling.