[Paper Review] Language Modeling with Gated Convolutional Networks

Yann Dauphin, Angela Fan+2 more|arXiv (Cornell University)|Dec 23, 2016

Topic Modeling24 references1,121 citations

TL;DR

Introduces gated convolutional networks (GCNN) with Gated Linear Units for language modeling, achieving state-of-the-art on WikiText-103 and competitive results on Google Billion Word with much higher efficiency than recurrent models.

ABSTRACT

The pre-dominant approach to language modeling to date is based on recurrent neural networks. Their success on this task is often linked to their ability to capture unbounded context. In this paper we develop a finite context approach through stacked convolutions, which can be more efficient since they allow parallelization over sequential tokens. We propose a novel simplified gating mechanism that outperforms Oord et al (2016) and investigate the impact of key architectural decisions. The proposed approach achieves state-of-the-art on the WikiText-103 benchmark, even though it features long-term dependencies, as well as competitive results on the Google Billion Words benchmark. Our model reduces the latency to score a sentence by an order of magnitude compared to a recurrent baseline. To our knowledge, this is the first time a non-recurrent approach is competitive with strong recurrent models on these large scale language tasks.

Motivation & Objective

• Motivate finite-context, highly parallelizable language modeling as an alternative to recurrent networks.
• Propose a gating mechanism (GLU) within gated convolutional nets to facilitate gradient flow and non-linear modeling.
• Evaluate GCNNs on large-scale datasets (Google Billion Word, WikiText-103) against recurrent models and baselines.
• Analyze the impact of context size, gating mechanisms, training techniques, and architectural choices on performance and efficiency.

Proposed method

• Define a convolutional architecture that uses gated linear units (GLU) within residual bottleneck blocks.
• Compute hidden representations via h_l(X) = (X * W + b) ⊗ σ(X * V + c) with appropriate causal padding to prevent peeking into the future.
• Use pre-activation residual blocks to enable deep stacking and gradient flow.
• Train with adaptive softmax for efficient large-vocabulary prediction.
• Experiment with gating variants (GLU vs GTU) and compare against linear and bilinear/non-linear alternatives.
• Evaluate context size effects and compare throughput and responsiveness against LSTMs.

Experimental results

Research questions

• RQ1 Can gated convolutional networks capture long-range dependencies in language modeling effectively without recurrence?
• RQ2 How does the GLU gating mechanism compare to LSTM-style gates in terms of training efficiency and perplexity?
• RQ3 What is the impact of context window size on performance for GCNNs on large-scale datasets?
• RQ4 How do architectural choices (residual blocks, bottlenecks) and optimization tricks affect training speed and final performance?
• RQ5 Are GCNNs competitive with state-of-the-art recurrent models on large language modeling benchmarks under realistic compute constraints?

Key findings

• GCNNs achieve state-of-the-art perplexity on WikiText-103 and competitive results on Google Billion Word.
• GLU-based GCNNs converge faster and to lower perplexities than LSTM-style gates and other activations on both Wik-103 and GBW.
• Context size improves performance up to around 20-40 tokens, with diminishing returns beyond that, even for long documents.
• Bottleneck residual blocks and weight normalization with gradient clipping substantially speed up training and improve efficiency.
• GCNNs offer strong throughput and superior responsiveness on GPUs, with significant speedups over recurrent baselines under comparable hardware and output-approximation settings.