[Paper Review] Challenges and Applications of Large Language Models

Jean Kaddour, Joshua Harris+4 more|arXiv (Cornell University)|Jul 19, 2023

Topic ModelingComputer Science172 citations

TL;DR

A survey that systematically outlines open problems (design, behavior, science) and current successes in applying large language models across diverse domains, highlighting constraints and trade-offs.

ABSTRACT

Large Language Models (LLMs) went from non-existent to ubiquitous in the machine learning discourse within a few years. Due to the fast pace of the field, it is difficult to identify the remaining challenges and already fruitful application areas. In this paper, we aim to establish a systematic set of open problems and application successes so that ML researchers can comprehend the field's current state more quickly and become productive.

Motivation & Objective

• Identify unresolved challenges in LLM design, behavior, and scientific progress.
• Catalog successful application domains and how challenges constrain them.
• Provide guidance for ML researchers to accelerate progress in LLM research and deployment.

Proposed method

• Classify challenges into three broad categories: design, behavior, and science.
• Review literature and reported techniques addressing each challenge.
• Summarize application areas and associated constraints to guide future work.

Figure 1: Overview of LLM Challenges. Design ing LLMs relates to decisions taken before deployment. Behavior ial challenges occur during deployment. Science challenges hinder academic progress.

Experimental results

Research questions

• RQ1 What challenges remain unresolved for large language models across design, behavior, and science?
• RQ2 Where are LLMs currently applied, and what limitations do these challenges impose on these applications?
• RQ3 What data, tokenization, training, fine-tuning, and evaluation practices influence LLM performance and trustworthiness?

Key findings

• Datasets for pre-training are vast and often unfathmanable, with near-duplicates and benchmark contamination impacting model behavior and evaluation.
• Tokenization and tokenizer–model coupling introduce language and resource inequities, especially for multilingual and low-resource languages.
• Pre-training costs are extremely high, driving interest in scaling laws, compute-optimal strategies, and alternative training objectives to improve data efficiency.
• Fine-tuning LLMs faces practical barriers due to memory and storage requirements, prompting exploration of parameter-efficient fine-tuning methods like adapters, prefix-tuning, and prompt-tuning.
• A variety of pre-training objectives (MLM, prefix LM, span denoising, MoD) and data construction strategies impact data efficiency and downstream transfer, with ongoing research into their trade-offs.
• The paper also surveys a wide range of applications including chatbots, computational biology, programming, creative work, knowledge work, law, medicine, reasoning, robotics, social sciences, and synthetic data generation.

Figure 2: Exemplary Drawbacks of relying on Tokenization. (1) The tokenizer training step involves non-trivial computations, e.g., multiple passes over the entire pre-training dataset, and introduces a dependency on it, which can become especially problematic in multilingual settings. (2) The embedd