[Paper Review] Language Models with Transformers

Chenguang Wang, Mu Li+1 more|arXiv (Cornell University)|Apr 20, 2019

Topic Modeling34 references46 citations

TL;DR

The paper proposes Coordinate Architecture Search (CAS) to adapt pre-trained GPT/BERT-like transformers for language modeling by adding LSTM layers and selective fine-tuning, achieving significant perplexity improvements over state-of-the-art LSTMs on PTB, WikiText-2, and WikiText-103.

ABSTRACT

The Transformer architecture is superior to RNN-based models in computational efficiency. Recently, GPT and BERT demonstrate the efficacy of Transformer models on various NLP tasks using pre-trained language models on large-scale corpora. Surprisingly, these Transformer architectures are suboptimal for language model itself. Neither self-attention nor the positional encoding in the Transformer is able to efficiently incorporate the word-level sequential context crucial to language modeling. In this paper, we explore effective Transformer architectures for language model, including adding additional LSTM layers to better capture the sequential context while still keeping the computation efficient. We propose Coordinate Architecture Search (CAS) to find an effective architecture through iterative refinement of the model. Experimental results on the PTB, WikiText-2, and WikiText-103 show that CAS achieves perplexities between 20.42 and 34.11 on all problems, i.e. on average an improvement of 12.0 perplexity units compared to state-of-the-art LSTMs. The source code is publicly available.

Motivation & Objective

• Motivate the need for architectures that capture fine-grained word-level sequential context in language modeling.
• Propose Coordinate Architecture Search (CAS) to efficiently refine pre-trained Transformer architectures for language modeling.
• Demonstrate that adding LSTM layers and selective fine-tuning improves perplexity over LSTM-based baselines.
• Showcase how CAS leverages prior GPT/BERT models to achieve strong results with reduced search cost.

Proposed method

• Introduce a Transformer-based language model augmented with LSTM layers to capture fine-grained sequential context.
• Define and apply Coordinate Architecture Search (CAS), a greedy, random-sampling procedure to generate and evaluate architecture variants.
• Fine-tune only a subset of weights or transform the architecture while reusing pre-trained GPT/BERT weights to reduce training cost.
• Evaluate CAS variants on PTB, WT-2, and WT-103 using sub-word vocabularies (WordPiece and BPE) and perplexity as the evaluation metric.
• Compare CAS against GPT, BERT, and AWD-LSTM-MoS baselines, including ablations without LSTMs or with fixed Transformer blocks.
• Report efficiency metrics contrasting CAS with NAS/ENAS/DARTS in terms of GPU-days and describe vocabulary and training data considerations.

Experimental results

Research questions

• RQ1 Can adding LSTM layers to Transformer-based language models improve word-level sequential context modeling for next-word prediction?
• RQ2 Does a greedy, architecture-refinement search (CAS) effectively identify high-performing language-model architectures from pre-trained GPT/BERT starting points?
• RQ3 How does CAS compare to standard LSTM/Transformer baselines and modern language models in perplexity on PTB, WT-2, and WT-103?
• RQ4 What is the impact of fine-tuning strategy (subset fixing vs full fine-tuning) on performance and overfitting across dataset sizes?

Key findings

• CAS-derived architectures (BERT-CAS and GPT-CAS) outperform AWD-LSTM-MoS baselines on PTB, WT-2, and WT-103 in perplexity.
• BERT-Large-CAS achieves strong perplexities—on PTB: 31.34 (val) and 34.11 (test on WT-103), illustrating benefits of larger pre-trained starting points.
• GPT-CAS achieves competitive perplexities, notably 34.24 test on WT-103 with sub-word vocabulary.
• Ablation studies show adding LSTMs and partially updating Transformer weights substantially improves performance over naive GPT/BERT usage.
• CAS demonstrates higher search efficiency (lower GPU-days) compared to NAS/ENAS/DARTS in the reported experiments.
• Compared at similar parameter counts, BERT-Large-CAS often surpasses GPT-2 on PTB and WT-103, illustrating favorable efficiency and effectiveness.