[Paper Review] Reducing the Barriers to Entry for Foundation Model Training

Paolo Faraboschi, Ellis Giles+3 more|arXiv (Cornell University)|Apr 12, 2024

Hip disorders and treatmentsMedicine3 citations

TL;DR

This paper proposes a fundamental shift in AI training infrastructure—spanning hardware, software, and system-level co-design—to drastically reduce the cost and energy demands of training foundation models. By leveraging analog in-memory computing and energy-based models, the authors demonstrate a pathway to lower barriers to entry, enabling broader access and fostering open innovation in large language model development.

ABSTRACT

The world has recently witnessed an unprecedented acceleration in demands for Machine Learning and Artificial Intelligence applications. This spike in demand has imposed tremendous strain on the underlying technology stack in supply chain, GPU-accelerated hardware, software, datacenter power density, and energy consumption. If left on the current technological trajectory, future demands show insurmountable spending trends, further limiting market players, stifling innovation, and widening the technology gap. To address these challenges, we propose a fundamental change in the AI training infrastructure throughout the technology ecosystem. The changes require advancements in supercomputing and novel AI training approaches, from high-end software to low-level hardware, microprocessor, and chip design, while advancing the energy efficiency required by a sustainable infrastructure. This paper presents the analytical framework that quantitatively highlights the challenges and points to the opportunities to reduce the barriers to entry for training large language models.

Motivation & Objective

• Address the unsustainable rise in costs and energy consumption associated with training large foundation models.
• Reduce the current market concentration in AI model development, which limits innovation and widens the technology gap.
• Enable broader participation in foundation model training by lowering computational and financial barriers.
• Explore revolutionary hardware and algorithmic approaches to achieve drastic reductions in training cost and energy use.
• Promote open innovation by making large-scale LLM training accessible to more organizations and researchers.

Proposed method

• Propose a co-design framework integrating supercomputing, accelerated computing, and energy-efficient hardware from microprocessor to datacenter scale.
• Introduce analog in-memory computing (IMC) to perform computations directly in memory, eliminating costly digital-to-analog conversions and reducing energy use.
• Utilize non-volatile nanoscale memories (e.g., memristor ReRAM) for inference and analog CMOS for training, enabling scalable, low-power operations.
• Apply energy-based models (EBMs), particularly quantum-inspired Hopfield Neural Networks, to reframe LLM training as an optimization problem in a native symbolic space.
• Use mixed-signal memory circuits and novel analog error-correction techniques to scale precision and support complex, high-order interactions in model training.
• Reframe token representation to preserve semantic relationships without expanding input space, reducing dependency on parameter-heavy architectures.

Experimental results

Research questions

• RQ1 What systemic changes in hardware, software, and system architecture are needed to make foundation model training affordable and sustainable?
• RQ2 How can analog in-memory computing reduce the energy and latency bottlenecks in large-scale LLM training?
• RQ3 Can energy-based models with programmable high-order interactions improve training efficiency and reduce computational cost?
• RQ4 To what extent can symbolic token representation in a native space eliminate the need for massive parameterization in LLMs?
• RQ5 What are the potential scalability and precision trade-offs in using mixed-signal, analog-based systems for end-to-end LLM training?

Key findings

• The current trajectory of LLM training growth—doubling parameters every 4 months—leads to unsustainable cost and energy consumption, threatening market concentration.
• Analog in-memory computing can significantly reduce energy use and latency by performing computations directly in memory, especially when using non-volatile nanoscale memories for inference and analog CMOS for training.
• Energy-based models, particularly quantum-inspired Hopfield networks, can model complex, high-order interactions in a compact circuit architecture, reducing the solution search space exponentially.
• By representing tokens symbolically and preserving their relationships without expanding input space, the model can reduce dependence on large parameter counts, lowering computational overhead.
• The integration of mixed-signal memory circuits and analog error-correction techniques enables scalable, high-precision training suitable for real-world LLM workloads.
• A co-design approach across software, algorithms, and hardware is essential to achieve drastic cost and energy reductions, with the potential to democratize access to foundation model training.