[Paper Review] Do Generative Large Language Models need billions of parameters?

Sia Gholami, Marwan Omar|arXiv (Cornell University)|Sep 12, 2023

Topic Modeling4 citations

TL;DR

This paper proposes GPT-Efficio, a highly efficient large language model that reduces parameter count through embedding layer factorization and cross-layer parameter sharing, achieving competitive performance with significantly fewer parameters—demonstrating that billion-parameter models are not essential for strong generative capabilities.

ABSTRACT

This paper presents novel systems and methodologies for the development of efficient large language models (LLMs). It explores the trade-offs between model size, performance, and computational resources, with the aim of maximizing the efficiency of these AI systems. The research explores novel methods that allow different parts of the model to share parameters, reducing the total number of unique parameters required. This approach ensures that the model remains compact without sacrificing its ability to learn and represent complex language structures. This study provides valuable insights and tools for creating more efficient and effective LLMs, contributing to a more sustainable and accessible future for AI language modeling.

Motivation & Objective

• To investigate whether generative large language models truly require billions of parameters to achieve high performance.
• To reduce computational and energy costs associated with training and deploying large models by minimizing parameter count.
• To improve model efficiency, accessibility, and sustainability through novel parameter sharing and factorization techniques.
• To enable deployment of advanced NLP capabilities on resource-constrained devices and in low-resource settings.
• To balance model expressivity and complexity by strategically sharing parameters across layers and embedding components.

Proposed method

• Applies embedding layer factorization to decompose high-dimensional embeddings into lower-rank matrices, reducing parameter count in the input projection layer.
• Implements parameter sharing across multiple transformer layers, where shared weights are tied across different layers to reduce unique parameter count.
• Uses a shared attention and feed-forward network structure across layers, minimizing redundancy in model architecture.
• Employs a modified training objective that maintains performance despite reduced parameter count, with careful initialization and regularization.
• Introduces a mechanism to control the extent of parameter sharing based on model depth and task complexity.
• Validates the approach through ablation studies comparing shared vs. unique parameters across various NLP benchmarks.

Experimental results

Research questions

• RQ1 Can a generative transformer model maintain strong performance with significantly fewer parameters through strategic parameter sharing?
• RQ2 How does embedding layer factorization affect model capacity and inference quality in large language models?
• RQ3 What trade-offs exist between model efficiency, training stability, and performance when sharing parameters across layers?
• RQ4 To what extent can parameter sharing reduce computational and energy costs without sacrificing generalization or controllability?
• RQ5 In which NLP tasks does parameter sharing provide the most significant efficiency gains without performance degradation?

Key findings

• The proposed GPT-Efficio model achieves competitive performance on standard NLP benchmarks despite having substantially fewer parameters than standard billion-parameter models.
• Embedding layer factorization reduces the number of parameters in the input projection layer by up to 70% without significant performance drop.
• Cross-layer parameter sharing reduces total unique parameters by approximately 40-50% while maintaining competitive accuracy on text generation and question-answering tasks.
• The model demonstrates faster training convergence and lower inference latency compared to baseline models with equivalent architecture but no parameter sharing.
• Empirical results show that parameter sharing improves generalization on low-resource and out-of-distribution data, reducing overfitting risks.
• Ablation studies confirm that parameter sharing is most effective when applied to attention and feed-forward sublayers, with diminishing returns when applied to all components.