[Paper Review] GrowLength: Accelerating LLMs Pretraining by Progressively Growing Training Length

Hongye Jin, Xiaotian Han+4 more|arXiv (Cornell University)|Oct 1, 2023

Topic Modeling4 citations

TL;DR

This paper proposes GrowLength, a method that accelerates Large Language Model (LLM) pretraining by progressively increasing the input sequence length from 128 to 4096 tokens during training. By starting with shorter sequences and gradually scaling up, the method improves training efficiency and model performance without additional engineering, enabling models to process more tokens in the same time and achieve lower training loss compared to fixed-length baselines.

ABSTRACT

The evolving sophistication and intricacies of Large Language Models (LLMs) yield unprecedented advancements, yet they simultaneously demand considerable computational resources and incur significant costs. To alleviate these challenges, this paper introduces a novel, simple, and effective method named ``\growlength'' to accelerate the pretraining process of LLMs. Our method progressively increases the training length throughout the pretraining phase, thereby mitigating computational costs and enhancing efficiency. For instance, it begins with a sequence length of 128 and progressively extends to 4096. This approach enables models to process a larger number of tokens within limited time frames, potentially boosting their performance. In other words, the efficiency gain is derived from training with shorter sequences optimizing the utilization of resources. Our extensive experiments with various state-of-the-art LLMs have revealed that models trained using our method not only converge more swiftly but also exhibit superior performance metrics compared to those trained with existing methods. Furthermore, our method for LLMs pretraining acceleration does not require any additional engineering efforts, making it a practical solution in the realm of LLMs.

Motivation & Objective

• To address the high computational cost and long training time of pretraining large language models.
• To explore whether dynamic sequence length during pretraining can improve training efficiency and model performance.
• To adapt context window extension techniques—successful in fine-tuning—into the pretraining phase.
• To develop a simple, effective, and plug-and-play method that enhances resource utilization during LLM pretraining.
• To demonstrate that training with progressively growing sequence lengths leads to faster convergence and better generalization.

Proposed method

• The method progressively increases the input sequence length during pretraining, starting from 128 tokens and growing to 4096 over time.
• Training begins with shorter sequences to maximize computational throughput and resource utilization in early stages.
• The sequence length increases in stages (e.g., 128 → 256 → 512 → 1024 → 2048 → 4096), allowing models to learn long-range dependencies gradually.
• The approach is orthogonal to existing acceleration techniques like quantization, pruning, and FlashAttention, and can be combined with them.
• The training ratio for each sequence length is heuristically selected to control the proportion of time spent at each length.
• The method leverages the insight that shorter sequences are computationally cheaper and that models can generalize to longer contexts after exposure to increasing sequence lengths.

Experimental results

Research questions

• RQ1 Can progressively growing the input sequence length during LLM pretraining accelerate training and improve model performance?
• RQ2 Does training with shorter sequences first and gradually increasing length lead to better token efficiency and faster convergence?
• RQ3 How sensitive is the method to the choice of sequence length progression schedule and ratios?
• RQ4 Can the method outperform fixed-sequence-length pretraining in terms of training loss and downstream performance within the same training time?
• RQ5 To what extent does GrowLength improve position extrapolation and long-context generalization?

Key findings

• Models trained with GrowLength achieve lower training loss than fixed-sequence baselines (e.g., LLM128 and LLM1024) when trained for the same duration.
• The method enables models to process significantly more tokens in the same training time, improving data efficiency.
• GrowLength outperforms both short-sequence (128) and long-sequence (1024) fixed-length baselines in terms of convergence speed and final performance.
• The method is robust to different ratios of sequence length progression, showing minimal performance drop even when intermediate lengths like 256 are omitted.
• A large jump in sequence length (e.g., from 128 to 1024) causes a sharp rise in training loss and performance degradation, indicating sensitivity to abrupt changes.
• GrowLength is the first method to accelerate LLM pretraining from the input sequence length perspective, offering a plug-and-play solution compatible with existing acceleration techniques.