[Paper Review] Optimizing Deeper Transformers on Small Datasets

Peng Xu, Dhruv Kumar+7 more|arXiv (Cornell University)|Dec 30, 2020

Topic Modeling35 references4 citations

TL;DR

This paper proposes DT-Fixup, a data-dependent weight initialization scheme that enables training very deep transformers (up to 48 layers) from scratch on small datasets without learning rate warm-up or layer normalization. By applying DT-Fixup to a hybrid model with 24 fine-tuned RoBERTa layers and 24 relation-aware layers trained from scratch, the method achieves state-of-the-art 70.9% exact match on the Spider Text-to-SQL benchmark with fewer training steps and no task-specific pre-training.

ABSTRACT

It is a common belief that training deep transformers from scratch requires large datasets. Consequently, for small datasets, people usually use shallow and simple additional layers on top of pre-trained models during fine-tuning. This work shows that this does not always need to be the case: with proper initialization and optimization, the benefits of very deep transformers can carry over to challenging tasks with small datasets, including Text-to-SQL semantic parsing and logical reading comprehension. In particular, we successfully train $48$ layers of transformers, comprising $24$ fine-tuned layers from pre-trained RoBERTa and $24$ relation-aware layers trained from scratch. With fewer training steps and no task-specific pre-training, we obtain the state-of-the-art performance on the challenging cross-domain Text-to-SQL parsing benchmark Spider. We achieve this by deriving a novel Data-dependent Transformer Fixed-update initialization scheme (DT-Fixup), inspired by the prior T-Fixup work. Further error analysis shows that increasing depth can help improve generalization on small datasets for hard cases that require reasoning and structural understanding.

Motivation & Objective

• To address the common belief that training deep transformers from scratch requires large datasets, especially for reasoning-intensive tasks with limited data.
• To overcome optimization challenges in training deep transformers on small datasets, such as instability from restricted batch sizes and reliance on learning rate warm-up or layer normalization.
• To develop a generalizable initialization method that enables deeper transformer architectures to improve generalization and reasoning ability on small, complex NLP benchmarks.
• To demonstrate that deeper transformers, when properly initialized, can outperform shallow models even without task-specific pre-training on datasets like Spider and ReCola.

Proposed method

• DT-Fixup derives a data-dependent initialization scheme inspired by T-Fixup, adapting weight scaling to the input data distribution to stabilize training without layer normalization or learning rate warm-up.
• The method extends T-Fixup to mixed architectures: combining pre-trained RoBERTa layers with newly trained relation-aware transformer layers, enabling deeper models on small datasets.
• It applies a modified weight initialization that accounts for the variance of input representations, ensuring stable gradients during backpropagation in deep networks.
• The approach is generalized to relation-aware transformers, allowing improved modeling of structural and relational dependencies in tasks like Text-to-SQL parsing.
• Training is conducted with smaller batch sizes typical of small datasets, yet DT-Fixup maintains convergence and generalization performance.
• The method is evaluated on two challenging benchmarks: Spider (cross-domain Text-to-SQL) and ReCola (logical reading comprehension), both with <10k training samples.

Experimental results

Research questions

• RQ1 Can very deep transformers (up to 48 layers) be successfully trained from scratch on small datasets without pre-training or large-batch training?
• RQ2 Does a data-dependent initialization scheme like DT-Fixup outperform standard training procedures in terms of generalization and convergence speed on small, complex NLP tasks?
• RQ3 To what extent do deeper transformer architectures improve performance on reasoning-intensive tasks like Text-to-SQL parsing and logical reading comprehension?
• RQ4 Why do standard training procedures fail on small datasets, and can optimization improvements alone restore the benefits of depth?

Key findings

• The proposed DT-Fixup method enables training a 48-layer transformer model (24 pre-trained RoBERTa layers + 24 relation-aware layers trained from scratch) on the Spider benchmark with only 10k training samples.
• The model achieves 70.9% exact match accuracy on the Spider test set, setting a new state-of-the-art result at the time of writing.
• DT-Fixup outperforms standard training with better generalization, especially on hard examples requiring reasoning and structural understanding.
• Error analysis shows that the deeper model (N=24) reduces sketch and both errors significantly compared to shallow and standard-trained models, indicating improved structural reasoning.
• The method eliminates the need for learning rate warm-up and layer normalization, enabling stable training with smaller batch sizes on small datasets.
• The improvement is not due to architectural design alone but stems from the optimization strategy, as deeper models with standard training fail to generalize on small data.