[Paper Review] The Evolved Transformer

David R. So, Liang Chen+1 more|arXiv (Cornell University)|Jan 30, 2019

Magnetic Properties and Applications196 citations

TL;DR

The paper applies evolutionary neural architecture search with Progressive Dynamic Hurdles, seeded by the Transformer, to find a faster and more accurate feed-forward seq2seq model that surpasses the Transformer on multiple language tasks. It achieves a new state-of-the-art BLEU on WMT’14 En-De and is more parameter-efficient at smaller sizes.

ABSTRACT

Recent works have highlighted the strength of the Transformer architecture on sequence tasks while, at the same time, neural architecture search (NAS) has begun to outperform human-designed models. Our goal is to apply NAS to search for a better alternative to the Transformer. We first construct a large search space inspired by the recent advances in feed-forward sequence models and then run evolutionary architecture search with warm starting by seeding our initial population with the Transformer. To directly search on the computationally expensive WMT 2014 English-German translation task, we develop the Progressive Dynamic Hurdles method, which allows us to dynamically allocate more resources to more promising candidate models. The architecture found in our experiments -- the Evolved Transformer -- demonstrates consistent improvement over the Transformer on four well-established language tasks: WMT 2014 English-German, WMT 2014 English-French, WMT 2014 English-Czech and LM1B. At a big model size, the Evolved Transformer establishes a new state-of-the-art BLEU score of 29.8 on WMT'14 English-German; at smaller sizes, it achieves the same quality as the original "big" Transformer with 37.6% less parameters and outperforms the Transformer by 0.7 BLEU at a mobile-friendly model size of 7M parameters.

Motivation & Objective

• Motivate neural architecture search (NAS) for improving feed-forward seq2seq models beyond the Transformer.
• Construct a large, Transformer-representative search space that includes modern seq2seq components.
• Develop Progressive Dynamic Hurdles (PDH) to efficiently search directly on compute-intensive tasks.
• Seed the search with the Transformer to improve search efficiency and performance.
• Demonstrate that the evolved architecture, the Evolved Transformer (ET), outperforms the Transformer across multiple tasks and sizes.

Proposed method

• Use tournament-selection evolutionary NAS with a gene-encoding that represents encoder/decoder blocks.
• Seed the initial population with the Transformer to anchor the search.
• Construct a two-cell search space (encoder and decoder) with NASNet-style blocks and multiple branch-level fields.
• Introduce Progressive Dynamic Hurdles (PDH) to allocate more training steps to promising candidates while discarding poor ones early.
• Train candidate models on WMT’14 En-De to evaluate fitness via validation perplexity, then mutate and select to evolve architectures.

Experimental results

Research questions

• RQ1 Can neural architecture search find a feed-forward seq2seq architecture superior to the Transformer for translation and language modeling?
• RQ2 Does seeding the search with the Transformer and using PDH improve NAS efficiency and final model quality?
• RQ3 What architectural characteristics emerge in the evolved model compared with the Transformer?
• RQ4 How does the Evolved Transformer (ET) compare to the Transformer across multiple tasks and model sizes?

Key findings

• ET consistently outperforms the Transformer across translation and language modeling tasks.
• On WMT’14 En-De, ET achieves a state-of-the-art BLEU of 29.8 with a comparable parameter count to the Transformer.
• At mobile-friendly sizes (~7M parameters), ET matches Transformer quality with 37.6% fewer parameters and gains ~0.7 BLEU.
• ET shows improvements at base and big sizes across En-De, En-Fr, En-Cs, and LM1B, with large gains in smaller models.
• ET’s notable architectural traits include wide depth-wise separable convolutions in lower layers, branching structures, gated activations, and swish activations.