[Paper Review] Trellis Networks for Sequence Modeling

Shaojie Bai, J. Zico Kolter+1 more|arXiv (Cornell University)|Oct 15, 2018

Topic ModelingComputer Science59 references68 citations

TL;DR

TrellisNet is a new sequence model that ties weights across layers and injects input into every layer, acting as a bridge between recurrent and convolutional models; it achieves state-of-the-art results on multiple language modeling and long-range benchmarks.

ABSTRACT

We present trellis networks, a new architecture for sequence modeling. On the one hand, a trellis network is a temporal convolutional network with special structure, characterized by weight tying across depth and direct injection of the input into deep layers. On the other hand, we show that truncated recurrent networks are equivalent to trellis networks with special sparsity structure in their weight matrices. Thus trellis networks with general weight matrices generalize truncated recurrent networks. We leverage these connections to design high-performing trellis networks that absorb structural and algorithmic elements from both recurrent and convolutional models. Experiments demonstrate that trellis networks outperform the current state of the art methods on a variety of challenging benchmarks, including word-level language modeling and character-level language modeling tasks, and stress tests designed to evaluate long-term memory retention. The code is available at https://github.com/locuslab/trellisnet .

Motivation & Objective

• Explore a new architecture (TrellisNet) for sequence modeling that unifies recurrent and convolutional approaches.
• Demonstrate that TrellisNet can reproduce truncated RNNs and outperform them on standard benchmarks.
• Show empirical gains on word-level and character-level language modeling and long-range memory tasks.
• Analyze connections between TrellisNet, TCNs, and RNNs to enable cross-pollination of techniques.

Proposed method

• Define TrellisNet as a causal, deep, weight-tied 1D convolutional architecture with input injection across layers.
• Show equivalence between M-truncated RNNs and TrellisNet with sparse interlayer kernels (Theorem 1).
• Use mixed group convolutions to realize sparse weight structure that mimics RNNs.
• Adopt gated activations inspired by LSTM cells within TrellisNet (gated activation).
• Leverage techniques from both CNNs (dilations, auxiliary losses, weight normalization) and RNNs (LSTM gates, variational dropout) to enhance TrellisNet.
• Evaluate on PTB, WT103, PTB character modeling, and long-range tasks (Sequential MNIST, Permuted MNIST, Sequential CIFAR-10).

Experimental results

Research questions

• RQ1 Can TrellisNet generalize truncated RNNs and serve as a bridge between RNNs and CNNs?
• RQ2 Does weight tying across depth and input injection improve sequence modeling performance?
• RQ3 Can TrellisNet achieve state-of-the-art perplexities for word-level and character-level language modeling?
• RQ4 How does TrellisNet perform on long-range memory benchmarks compared to recurrent and self-attention models?

Key findings

• TrellisNet sets new state-of-the-art perplexities on Penn Treebank word-level modeling and WikiText-103.
• On PTB, TrellisNet with 24M parameters achieves 56.97 perplexity, with larger configurations reaching 54.19.
• On WT103, TrellisNet achieves 29.19 perplexity, outperforming Relational Memory Core and Merity et al. results.
• For character-level PTB, TrellisNet achieves 1.158 bits-per-character, surpassing prior bests.
• On long-range benchmarks, TrellisNet attains the highest reported accuracies: Seq MNIST 99.20, Permuted MNIST 98.13, Seq CIFAR-10 73.42.