[Paper Review] A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications

Pranab Sahoo, Ayush Singh+4 more|arXiv (Cornell University)|Feb 5, 2024

Topic Modeling183 citations

TL;DR

A systematic survey organizing prompt engineering techniques for LLMs and VLMs by application area, detailing methods, models, datasets, and pros/cons across 29+ approaches.

ABSTRACT

Prompt engineering has emerged as an indispensable technique for extending the capabilities of large language models (LLMs) and vision-language models (VLMs). This approach leverages task-specific instructions, known as prompts, to enhance model efficacy without modifying the core model parameters. Rather than updating the model parameters, prompts allow seamless integration of pre-trained models into downstream tasks by eliciting desired model behaviors solely based on the given prompt. Prompts can be natural language instructions that provide context to guide the model or learned vector representations that activate relevant knowledge. This burgeoning field has enabled success across various applications, from question-answering to commonsense reasoning. However, there remains a lack of systematic organization and understanding of the diverse prompt engineering methods and techniques. This survey paper addresses the gap by providing a structured overview of recent advancements in prompt engineering, categorized by application area. For each prompting approach, we provide a summary detailing the prompting methodology, its applications, the models involved, and the datasets utilized. We also delve into the strengths and limitations of each approach and include a taxonomy diagram and table summarizing datasets, models, and critical points of each prompting technique. This systematic analysis enables a better understanding of this rapidly developing field and facilitates future research by illuminating open challenges and opportunities for prompt engineering.

Motivation & Objective

• Provide a structured, application-centric taxonomy of prompt engineering techniques for LLMs and VLMs.
• Summarize prompting methodologies, applications, models, and datasets for each technique.
• Evaluate strengths, limitations, and open challenges to guide future research.

Proposed method

• Organize techniques by application area (new tasks, reasoning, reduce hallucinations, user interface, fine-tuning, knowledge-based reasoning, etc.).
• Describe prompting methodologies (zero-shot, few-shot, CoT, Auto-CoT, Self-Consistency, ToT, GoT, S2A, CoVe, CoN, CoK, ReAct, RAG, CoT variants, etc.).
• Provide model/dataset mappings and performance highlights (e.g., CoT with PaLM 540B achieving 90.2% accuracy on math/commonsense benchmarks).
• Discuss strengths/limitations and practical considerations of each technique.
• Include a taxonomy diagram and a summary table mapping techniques to datasets, models, and key points.

Figure 1 : Visual breakdown of prompt engineering components: LLMs trained on extensive data, instruction and context as pivotal elements shaping the prompt, and a user input interface.

Experimental results

Research questions

• RQ1 What are the predominant prompt engineering techniques used across LLMs and VLMs?
• RQ2 How do prompting methods vary by application domain and task type?
• RQ3 What datasets and models are commonly used to evaluate prompting approaches, and what are their reported strengths and limitations?
• RQ4 What are the open challenges and opportunities for advancing prompt engineering in practice?

Key findings

• The survey catalogs over 29 prompting techniques spanning zero-shot to advanced methods like Tree-of-Thoughts and Graph-of-Thoughts.
• CoT prompting plus enhancements (Self-Consistency, ToT, GoT, LogiCoT, CoS, ReAct, CoVe, CoN, CoK) yield notable gains on multiple benchmarks (e.g., CoT with PaLM 540B achieves 90.2% accuracy on math/commonsense benchmarks).
• Retrieval-augmented and tool-use approaches (RAG, ReAct, CoVe, CoN, ART) address hallucinations and factuality, with demonstrated gains on QA, fact verification, and open-domain tasks.
• Code generation and execution prompting (Scratchpad, PoT, SCoT, CoC) improve reasoning and accuracy on programming and math tasks, outperforming CoT in several settings.
• There is a broad spectrum of techniques addressing uncertainty, emotion/tone, efficiency, metacognition, and user interaction, highlighting a mature, multi-faceted prompt engineering landscape.
• The paper provides a taxonomy diagram and a consolidated table outlining datasets, models, and prompts across techniques.