[Paper Review] Large Language Models: A Survey

Shervin Minaee, Tomas Mikolov+5 more|arXiv (Cornell University)|Feb 9, 2024

Topic Modeling210 citations

TL;DR

A comprehensive survey of large language models (LLMs), covering GPT, LLaMA, PaLM families, training/data, evaluation, augmentation, datasets, benchmarks, and open challenges.

ABSTRACT

Large Language Models (LLMs) have drawn a lot of attention due to their strong performance on a wide range of natural language tasks, since the release of ChatGPT in November 2022. LLMs' ability of general-purpose language understanding and generation is acquired by training billions of model's parameters on massive amounts of text data, as predicted by scaling laws \cite{kaplan2020scaling,hoffmann2022training}. The research area of LLMs, while very recent, is evolving rapidly in many different ways. In this paper, we review some of the most prominent LLMs, including three popular LLM families (GPT, LLaMA, PaLM), and discuss their characteristics, contributions and limitations. We also give an overview of techniques developed to build, and augment LLMs. We then survey popular datasets prepared for LLM training, fine-tuning, and evaluation, review widely used LLM evaluation metrics, and compare the performance of several popular LLMs on a set of representative benchmarks. Finally, we conclude the paper by discussing open challenges and future research directions.

Motivation & Objective

• Summarize the evolution of language models from statistical to transformer-based LLMs and their emergent abilities.
• Review three major LLM families (GPT, LLaMA, PaLM) and notable open-source/open-access models.
• Outline methodologies for building, fine-tuning, and augmenting LLMs, including RLHF and instruction tuning.
• Catalog datasets and evaluation metrics used for training, fine-tuning, and benchmarking LLMs.
• Highlight challenges and future research directions in LLM development and deployment.

Proposed method

• Survey the historical waves of language modeling (statistical, neural, pre-trained, LLMs).
• Characterize LLM capabilities such as in-context learning, instruction following, and multi-step reasoning.
• Compare GPT, LLaMA, and PaLM families, including variants and open-source efforts.
• Describe pre-training versus fine-tuning paradigms and augmentation techniques (RLHF, external tools).
• Summarize datasets and benchmarks used for training and evaluation of LLMs.

Figure 1 : LLM Capabilities.

Experimental results

Research questions

• RQ1 What are the key capabilities and limitations of current LLM families (GPT, LLaMA, PaLM)?
• RQ2 How do training data, model scale, and architecture influence LLM performance on benchmarks?
• RQ3 What augmentation and alignment techniques improve LLM usefulness, safety, and reliability?
• RQ4 What datasets and metrics are standard for evaluating LLMs across tasks?
• RQ5 What are the primary open challenges and future directions in LLM research?

Key findings

• LLMs demonstrate emergent abilities, including in-context learning, instruction following, and multi-step reasoning.
• Three prominent LLM families (GPT, LLaMA, PaLM) illustrate different trade-offs in openness, scale, and performance.
• Instruction tuning and RLHF data significantly influence alignment and task performance.
• A broad ecosystem of datasets and benchmarks supports training, fine-tuning, and evaluation of LLMs.
• A range of augmentation strategies (external tools, retrieval, and continuous learning) enables LLM-based agents.
• The field identifies open challenges and future research directions in scalability, safety, and generalization.

Figure 2 : The paper structure.