[Paper Review] DeLighT: Deep and Light-weight Transformer

Sachin Mehta, Marjan Ghazvininejad+3 more|arXiv (Cornell University)|Aug 3, 2020

Topic Modeling45 references46 citations

TL;DR

DeLighT is a deep, parameter-efficient transformer that uses a DeLighT transformation with group linear transformations and block-wise scaling to achieve similar or better performance than standard transformers with far fewer parameters and operations.

ABSTRACT

We introduce a deep and light-weight transformer, DeLighT, that delivers similar or better performance than standard transformer-based models with significantly fewer parameters. DeLighT more efficiently allocates parameters both (1) within each Transformer block using the DeLighT transformation, a deep and light-weight transformation, and (2) across blocks using block-wise scaling, which allows for shallower and narrower DeLighT blocks near the input and wider and deeper DeLighT blocks near the output. Overall, DeLighT networks are 2.5 to 4 times deeper than standard transformer models and yet have fewer parameters and operations. Experiments on benchmark machine translation and language modeling tasks show that DeLighT matches or improves the performance of baseline Transformers with 2 to 3 times fewer parameters on average. Our source code is available at: \url{https://github.com/sacmehta/delight}

Motivation & Objective

• Motivate a parameter-efficient sequence model that scales in depth and width without proportional parameter costs.
• Propose the DeLighT transformation using group linear transformations with feature shuffling to learn wider representations efficiently.
• Introduce block-wise scaling to allocate more capacity to deeper, wider blocks near the output while keeping input blocks shallow and narrow.
• Demonstrate that DeLighT achieves comparable or better performance than baselines on translation and language modeling with substantially fewer parameters and FLOPs.

Proposed method

• Introduce DeLighT transformation that expands a dm-dimensional input to a high-dimensional space using N GLT layers, then reduces to do dimensions.
• Replace multi-head attention and FFN in transformers with single-head attention and a light-weight FFN.
• Use feature shuffling to share information across GLT groups and an input mixer to stabilize training.
• Decouple depth and width from input size, enabling block-wise scaling where early blocks are shallower/narrow and later blocks are deeper/wider.
• Define block-wise scaling with Nmin, Nmax and wm to create DeLighT blocks of varying depth and width across the network.
• Show that do = dm/2 reduces attention cost by a factor of about 2, and that FFN is reduced by about 16x due to the light-weight design.

Experimental results

Research questions

• RQ1 Can a deep, light-weight transformer with width/depth decoupled from input size match or exceed transformer baselines with significantly fewer parameters?
• RQ2 Do block-wise scaling and DeLighT transformations yield better parameter utilization and results on machine translation and language modeling tasks compared to uniform scaling?
• RQ3 How does DeLighT compare to state-of-the-art transformers in terms of BLEU and perplexity with fewer parameters and FLOPs?

Key findings

• DeLighT achieves similar or better BLEU scores than baseline Transformers on WMT and IWSLT datasets with 2.8x to 1.8x fewer parameters.
• DeLighT is 2.5–4x deeper than standard transformers yet uses fewer parameters and operations.
• On WMT’14 En-Fr, DeLighT delivers +1.3 BLEU with 13M fewer parameters and 3B fewer operations compared to a Transformer baseline.
• In language modeling (WikiText-103), DeLighT matches Transformer-XL performance with 1.5x fewer parameters.
• Block-wise scaling allocates fewer parameters near the input and more deeper capacity near the output, improving efficiency without sacrificing performance.
• Regularization needs are reduced for DeLighT relative to baseline transformers, implying stronger representations from the transformation itself.