[Paper Review] Efficient Training of Large Language Models on Distributed Infrastructures: A Survey

Jiangfei Duan, Shuo Zhang+14 more|arXiv (Cornell University)|Jul 29, 2024

Topic Modeling4 citations

TL;DR

This survey provides a comprehensive analysis of distributed training systems for large language models (LLMs), covering infrastructure, parallelism, and optimization techniques for scalability, efficiency, and reliability. It synthesizes advances in AI accelerators, high-bandwidth networking, memory and communication optimizations, and fault tolerance—highlighting live migration and module redundancy for failure recovery—while advocating for optoelectronic integration as a future enabler for large-scale LLM training.

ABSTRACT

Large Language Models (LLMs) like GPT and LLaMA are revolutionizing the AI industry with their sophisticated capabilities. Training these models requires vast GPU clusters and significant computing time, posing major challenges in terms of scalability, efficiency, and reliability. This survey explores recent advancements in training systems for LLMs, including innovations in training infrastructure with AI accelerators, networking, storage, and scheduling. Additionally, the survey covers parallelism strategies, as well as optimizations for computation, communication, and memory in distributed LLM training. It also includes approaches of maintaining system reliability over extended training periods. By examining current innovations and future directions, this survey aims to provide valuable insights towards improving LLM training systems and tackling ongoing challenges. Furthermore, traditional digital circuit-based computing systems face significant constraints in meeting the computational demands of LLMs, highlighting the need for innovative solutions such as optical computing and optical networks.

Motivation & Objective

• Address the growing challenges in training massive LLMs, such as GPT and LLaMA, which require weeks to months on tens of thousands of GPUs.
• Identify key bottlenecks in distributed LLM training, including computation inefficiency, communication overhead, memory pressure, and system unreliability.
• Systematically review advancements in training infrastructure, including AI accelerators, high-performance networking, and storage systems tailored for LLM workloads.
• Analyze parallelism strategies (data, tensor, pipeline) and their impact on system efficiency and scalability.
• Explore optimization techniques for computation, communication, memory, and fault tolerance to improve model FLOPs utilization and training resilience.

Proposed method

• Survey and categorize recent innovations in distributed LLM training infrastructure, including GPU clusters, optical interconnects, and scalable storage systems.
• Examine parallelism schemes—data, tensor, and pipeline parallelism—along with their trade-offs in communication, memory, and load balancing.
• Investigate computation optimizations such as kernel fusion, mixed-precision training, and hardware-aware operator scheduling to boost MFU.
• Analyze memory optimization techniques like activation checkpointing, offloading, and parameter quantization to reduce GPU memory pressure.
• Review communication optimization strategies, including gradient compression, all-reduce optimizations, and topology-aware communication scheduling.
• Evaluate fault tolerance mechanisms such as live migration and module redundancy (e.g., Parcae, Oobleck, Bamboo, SlipStream, SWARM) for fast failure recovery without checkpoint reloads.

Figure 1 : Overall structure of this survey.

Experimental results

Research questions

• RQ1 How can distributed training systems achieve high scalability across tens of thousands of GPUs while maintaining model accuracy?
• RQ2 What are the most effective strategies for optimizing computation, communication, and memory in large-scale LLM training to maximize Model FLOPs Utilization (MFU)?
• RQ3 How can system reliability be maintained over extended training periods (weeks to months) in the face of hardware, network, and software failures?
• RQ4 What role can emerging technologies like optical computing and photonic interconnects play in overcoming the limitations of traditional digital circuit-based systems?
• RQ5 How do live migration and module redundancy enable faster failure recovery compared to traditional checkpoint-based recovery in LLM training?

Key findings

• LLaMA-3 training on 16,384 H100 GPUs takes approximately 54 days, highlighting the extreme computational demands and need for efficient distributed systems.
• Model FLOPs Utilization (MFU) is a key metric for efficiency, and optimizations in computation, communication, and memory are critical to improving it.
• Live migration techniques like those in Parcae and Oobleck enable fast recovery by dynamically reconfiguring training pipelines using existing model replicas.
• Module redundancy approaches such as Bamboo and SlipStream allow uninterrupted training by rerouting computation to redundant modules during failures.
• SWARM combines redundant computation with instance migration to support heterogeneous and unreliable clusters, improving fault tolerance in real-world deployments.
• Optical computing and photonic interconnects—such as in TopoOpt and TPUv4—show promise in enhancing communication efficiency and enabling future-scale LLM training beyond Moore’s Law limits.

Figure 2 : A typical Transformer layer contains an Attention block and a Feed-Forward Network (FFN) block.