[Paper Review] Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View

Yiping Lu, Zhuohan Li+6 more|arXiv (Cornell University)|Jun 6, 2019

Topic ModelingComputer Science46 references116 citations

TL;DR

The paper reinterprets Transformer as a numerical ODE solver for a multi-particle diffusion-convection system, then proposes Macaron Net using Strang-Marchuk splitting for improved performance over standard Transformer.

ABSTRACT

The Transformer architecture is widely used in natural language processing. Despite its success, the design principle of the Transformer remains elusive. In this paper, we provide a novel perspective towards understanding the architecture: we show that the Transformer can be mathematically interpreted as a numerical Ordinary Differential Equation (ODE) solver for a convection-diffusion equation in a multi-particle dynamic system. In particular, how words in a sentence are abstracted into contexts by passing through the layers of the Transformer can be interpreted as approximating multiple particles' movement in the space using the Lie-Trotter splitting scheme and the Euler's method. Given this ODE's perspective, the rich literature of numerical analysis can be brought to guide us in designing effective structures beyond the Transformer. As an example, we propose to replace the Lie-Trotter splitting scheme by the Strang-Marchuk splitting scheme, a scheme that is more commonly used and with much lower local truncation errors. The Strang-Marchuk splitting scheme suggests that the self-attention and position-wise feed-forward network (FFN) sub-layers should not be treated equally. Instead, in each layer, two position-wise FFN sub-layers should be used, and the self-attention sub-layer is placed in between. This leads to a brand new architecture. Such an FFN-attention-FFN layer is "Macaron-like", and thus we call the network with this new architecture the Macaron Net. Through extensive experiments, we show that the Macaron Net is superior to the Transformer on both supervised and unsupervised learning tasks. The reproducible codes and pretrained models can be found at https://github.com/zhuohan123/macaron-net

Motivation & Objective

• Provide a novel interpretation of Transformer through multi-particle dynamic system (MPDS) and ODE theory.
• Leverage numerical analysis (Lie-Trotter vs. Strang-Marchuk splitting) to design a more accurate neural architecture.
• Demonstrate that the Macaron Net outperforms standard Transformer on supervised and unsupervised NLP tasks.

Proposed method

• Model Transformer layers as an ODE solver for an MPDS with convection (FFN) and diffusion (self-attention).
• Use Lie-Trotter splitting to map layer stacking to Euler steps in time.
• Replace Lie-Trotter with Strang-Marchuk splitting to create a three-sub-layer Macaron+architecture (FFN-half, Attention, FFN-half).
• Define Macaron layer as FFN-attention-FFN with half-step residuals and full-step residuals accordingly.
• Construct Macaron Net by stacking Macaron layers with equivalent parameter count to Transformer baselines.
• Empirically evaluate on machine translation (IWSLT14 De-En, WMT14 En-De) and GLUE-style unsupervised pretraining (BERT-like).

Experimental results

Research questions

• RQ1 Can Transformer be understood as a numerical ODE solver for a convection-diffusion MPDS?
• RQ2 Does adopting a Strang-Marchuk splitting scheme improve accuracy and performance over Lie-Trotter in neural architectures?
• RQ3 Do Macaron layers (FFN-attention-FFN) yield better NLP performance under the same parameter budget?
• RQ4 How does Macaron Net perform on supervised translation and unsupervised pretraining tasks compared to Transformer?
• RQ5 What empirical gains arise from deeper integration of ODE-based design principles into attention-based NLP models?

Key findings

• Macaron Net achieves higher BLEU than Transformer on IWSLT14 De-En (35.4 vs 34.4 for small) and WMT14 En-De (28.9 base, 30.2 big).
• Macaron Net outperforms Transformer big on WMT14 En-De by 1.8 BLEU points according to Table 1.
• On GLUE, Macaron Net base outperforms BERT base and all baselines, achieving higher general GLUE scores.
• In supervised MT, Macaron small surpasses Transformer small by 1.0 BLEU on IWSLT14 De-En; in WMT14 En-De, Macaron base surpasses Transformer base by 1.6 BLEU points.
• Unsupervised pretraining with Macaron Net yields improved downstream task performance over the baseline BERT/Transformer configurations.
• Theoretical analysis predicts Strang-Marchuk splitting reduces local truncation error from O(γ^2) to O(γ^3), motivating the Macaron-layer design.