[Paper Review] Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond

Jingfeng Yang, Hongye Jin+6 more|arXiv (Cornell University)|Apr 26, 2023

Artificial Intelligence in Healthcare and EducationMedicine149 citations

TL;DR

The paper provides a practical guide for using Large Language Models (LLMs) in downstream NLP tasks, comparing LLMs with fine-tuned models, and outlining data-driven considerations, task-specific use cases, and deployment factors.

ABSTRACT

This paper presents a comprehensive and practical guide for practitioners and end-users working with Large Language Models (LLMs) in their downstream natural language processing (NLP) tasks. We provide discussions and insights into the usage of LLMs from the perspectives of models, data, and downstream tasks. Firstly, we offer an introduction and brief summary of current GPT- and BERT-style LLMs. Then, we discuss the influence of pre-training data, training data, and test data. Most importantly, we provide a detailed discussion about the use and non-use cases of large language models for various natural language processing tasks, such as knowledge-intensive tasks, traditional natural language understanding tasks, natural language generation tasks, emergent abilities, and considerations for specific tasks.We present various use cases and non-use cases to illustrate the practical applications and limitations of LLMs in real-world scenarios. We also try to understand the importance of data and the specific challenges associated with each NLP task. Furthermore, we explore the impact of spurious biases on LLMs and delve into other essential considerations, such as efficiency, cost, and latency, to ensure a comprehensive understanding of deploying LLMs in practice. This comprehensive guide aims to provide researchers and practitioners with valuable insights and best practices for working with LLMs, thereby enabling the successful implementation of these models in a wide range of NLP tasks. A curated list of practical guide resources of LLMs, regularly updated, can be found at \url{https://github.com/Mooler0410/LLMsPracticalGuide}.

Motivation & Objective

• Motivate practitioners to understand when to use LLMs versus fine-tuned models for NLP tasks.
• Explain how pre-training, fine-tuning, and test data shape LLM performance.
• Provide task-specific guidance for NLU, NLG, and knowledge-intensive tasks and identify limitations.
• Offer practical recommendations and a decision flow to aid deployment in real-world scenarios.

Proposed method

• Classifies LLMs into encoder-decoder, encoder-only, and decoder-only architectures and discusses their training paradigms (masked language modeling vs autoregressive).
• Defines practical distinctions between LLMs and fine-tuned models (LLMs pretrained on broad data; fine-tuned models pretrained then task-tuned) and discusses scale implications (<20B params as fine-tuned level).
• Analyzes data influence from pretraining, training/tuning data, and test data, including zero-, few-, and abundant-annotation regimes, with guidance on when to use LLMs vs fine-tuned models.
• Examines NLP tasks (NLU, NLG, knowledge-intensive tasks) and emergent abilities, providing use-case vs no-use-case guidance and highlighting limitations.
• Proposes a practical decision flow (Figure 3) to choose between LLMs and fine-tuned models for different tasks.

Experimental results

Research questions

• RQ1 When should practitioners choose LLMs over fine-tuned models for downstream NLP tasks?
• RQ2 How do pretraining data, fine-tuning data, and test data impact LLM performance in practice?
• RQ3 What are the practical use cases and limitations of LLMs across traditional NLU, generation, and knowledge-intensive tasks?
• RQ4 What deployment considerations (efficiency, cost, latency) influence the practical use of LLMs?

Key findings

• LLMs generalize better than fine-tuned models on out-of-distribution data and with limited annotated data.
• Fine-tuned models often outperform LLMs on traditional NLU tasks with rich annotated data and lower compute costs.
• LLMs excel at generation tasks, including open-ended text and code synthesis, and perform well on many knowledge-intensive tasks.
• Retrieval-augmented approaches can convert closed-book tasks into open-book ones, improving performance for knowledge-heavy tasks.
• Scaling drives emergent abilities and improved reasoning but does not guarantee uniform gains across all tasks.
• RLHF and instruction-following capabilities (e.g., InstructGPT, ChatGPT) improve generalization and robustness, including multilingual instruction adherence.