[Paper Review] A Comprehensive Overview of Large Language Models

Humza Naveed, Asad Ullah Khan+7 more|arXiv (Cornell University)|Jul 12, 2023

Topic Modeling351 citations

TL;DR

This paper provides a self-contained, comprehensive survey of large language models (LLMs), covering architectures, training, fine-tuning, multi-modal extensions, datasets, evaluation, efficiency, and future challenges. It also offers detailed summaries of prominent pre-trained LLMs and practical guidance for researchers and practitioners.

ABSTRACT

Large Language Models (LLMs) have recently demonstrated remarkable capabilities in natural language processing tasks and beyond. This success of LLMs has led to a large influx of research contributions in this direction. These works encompass diverse topics such as architectural innovations, better training strategies, context length improvements, fine-tuning, multi-modal LLMs, robotics, datasets, benchmarking, efficiency, and more. With the rapid development of techniques and regular breakthroughs in LLM research, it has become considerably challenging to perceive the bigger picture of the advances in this direction. Considering the rapidly emerging plethora of literature on LLMs, it is imperative that the research community is able to benefit from a concise yet comprehensive overview of the recent developments in this field. This article provides an overview of the existing literature on a broad range of LLM-related concepts. Our self-contained comprehensive overview of LLMs discusses relevant background concepts along with covering the advanced topics at the frontier of research in LLMs. This review article is intended to not only provide a systematic survey but also a quick comprehensive reference for the researchers and practitioners to draw insights from extensive informative summaries of the existing works to advance the LLM research.

Motivation & Objective

• Provide a concise, comprehensive overview of recent developments in Large Language Models (LLMs).
• Summarize architectural and training details of pre-trained LLMs with fine-grained information.
• Discuss fine-tuning, multi-modal LLMs, augmented LLMs, datasets, benchmarks, evaluation, and deployment considerations.

Proposed method

• Survey the LLM literature to present background, architecture, training pipelines, and strategies.
• Summarize prominent pre-trained LLMs with architecture and training details in tables.
• Discuss configuration, evaluation, datasets, benchmarks, and practical considerations for practitioners.

Figure 1: The trend of papers released over years containing keywords “Large Language Model”, “Large Language Model + Fine-Tuning”, and “Large Language Model + Alignment”.

Experimental results

Research questions

• RQ1 What are the key architectural choices and training strategies across major LLMs?
• RQ2 How do fine-tuning, instruction-tuning, and alignment-tuning affect zero-shot and few-shot performance?
• RQ3 What datasets, benchmarks, and evaluation methods are used to assess LLMs, and what are the identified challenges?
• RQ4 What are the efficiency, deployment, and safety considerations in LLM research and practice?

Key findings

• LLMs have evolved toward instruction-tuned and increasingly open-source models.
• Emergent abilities such as reasoning and in-context learning appear at large scales, influencing broad applications.
• Efficiency approaches (parameter-efficient tuning, pruning, quantization, MoE, and context length strategies) are actively studied to reduce cost.
• A wide range of datasets and benchmarks are used to evaluate LLMs, highlighting goals in factual accuracy and alignment with human preferences.
• Researchers are expanding LLMs to multi-modal and agent-oriented settings, including robotics and tool use.
• Challenges include factual accuracy, alignment with human values, safety, and resource-intensive training and inference.

Figure 2: Chronological display of LLM releases: light blue rectangles represent ‘pre-trained’ models, while dark rectangles correspond to ‘instruction-tuned’ models. Models on the upper half signify open-source availability, whereas those on the bottom half are closed-source. The chart illustrates