[Paper Review] DeLighT: Very Deep and Light-weight Transformer

Sachin Mehta, Marjan Ghazvininejad+3 more|arXiv (Cornell University)|Aug 3, 2020

Topic Modeling30 references22 citations

TL;DR

DeLighT is a very deep and lightweight Transformer architecture that achieves superior or comparable performance to standard Transformers with significantly fewer parameters and FLOPs. It uses DExTra for parameter-efficient block-wise transformations and adaptive block scaling to deepen and widen layers toward the output, resulting in models 2.5–4× deeper but with reduced parameter counts and improved efficiency on machine translation and language modeling tasks.

ABSTRACT

We introduce a very deep and light-weight transformer, DeLighT, that delivers similar or better performance than transformer-based models with significantly fewer parameters. DeLighT more efficiently allocates parameters both (1) within each Transformer block using DExTra, a deep and light-weight transformation and (2) across blocks using block-wise scaling, that allows for shallower and narrower DeLighT blocks near the input and wider and deeper DeLighT blocks near the output. Overall, DeLighT networks are 2.5 to 4 times deeper than standard transformer models and yet have fewer parameters and operations. Experiments on machine translation and language modeling tasks show that DeLighT matches the performance of baseline Transformers with significantly fewer parameters. On the WMT'14 En-Fr high resource dataset, DeLighT requires 1.8 times fewer parameters and 2 times fewer operations and achieves better performance (+0.4 BLEU score) than baseline transformers. On the WMT'16 En-Ro low resource dataset, DeLighT delivers similar performance with 2.8 times fewer parameters than baseline transformers.

Motivation & Objective

• To design a Transformer architecture that maintains or improves performance while drastically reducing model size and computational cost.
• To address the inefficiency of standard Transformers, which scale poorly in depth and parameter count despite high performance.
• To explore parameter allocation strategies that prioritize deeper and wider blocks in later layers for improved representational capacity.
• To develop a method that enables very deep networks with minimal parameter growth, enhancing efficiency without sacrificing accuracy.

Proposed method

• Introduces DExTra, a deep and lightweight transformation mechanism that restructures attention and feed-forward layers within each Transformer block to reduce parameter count.
• Applies block-wise scaling to vary the width and depth of Transformer blocks across the network, making earlier blocks narrower and shallower and later blocks wider and deeper.
• Uses a progressive architectural design where deeper and wider blocks are placed toward the output to capture complex representations more efficiently.
• Optimizes parameter distribution by focusing computational resources on later layers, where hierarchical abstraction is most beneficial.
• Employs a residual connection scheme that maintains training stability despite extreme depth and reduced width in early layers.
• Leverages knowledge distillation and self-supervised pretraining to enhance performance with minimal parameters.

Experimental results

Research questions

• RQ1 Can a Transformer be made significantly deeper while reducing parameter count and FLOPs without sacrificing performance?
• RQ2 How can parameter allocation within and across Transformer blocks be optimized for efficiency and representational capacity?
• RQ3 Does placing deeper and wider blocks toward the output improve model performance compared to uniform or shallow blocks?
• RQ4 Can a lightweight architecture like DeLighT outperform standard Transformers on low-resource and high-resource machine translation tasks with fewer parameters?
• RQ5 What is the impact of block-wise scaling and DExTra on training stability and inference efficiency?

Key findings

• On the WMT'14 En-Fr translation dataset, DeLighT achieves a +0.4 BLEU score improvement over standard Transformers while using 1.8 times fewer parameters and 2 times fewer FLOPs.
• On the low-resource WMT'16 En-Ro dataset, DeLighT matches baseline performance with 2.8 times fewer parameters.
• DeLighT models are 2.5 to 4 times deeper than standard Transformers, demonstrating that extreme depth can be achieved efficiently with parameter-aware architectural design.
• The combination of DExTra and block-wise scaling enables high representational capacity with minimal parameter growth, especially in deeper layers.
• The model maintains strong performance across both high-resource and low-resource settings, indicating robustness to data scarcity.
• The efficiency gains are consistent across both parameter count and FLOP reduction, highlighting DeLighT’s practical advantages for deployment.