[Paper Review] A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT

Jules White, Quchen Fu+7 more|arXiv (Cornell University)|Feb 21, 2023

Software Engineering Research771 citations

TL;DR

The paper presents a catalog of prompt patterns—organized as reusable solutions—that improve prompt engineering for ChatGPT, including a framework for documenting patterns and their combination for diverse tasks in software development.

ABSTRACT

Prompt engineering is an increasingly important skill set needed to converse effectively with large language models (LLMs), such as ChatGPT. Prompts are instructions given to an LLM to enforce rules, automate processes, and ensure specific qualities (and quantities) of generated output. Prompts are also a form of programming that can customize the outputs and interactions with an LLM. This paper describes a catalog of prompt engineering techniques presented in pattern form that have been applied to solve common problems when conversing with LLMs. Prompt patterns are a knowledge transfer method analogous to software patterns since they provide reusable solutions to common problems faced in a particular context, i.e., output generation and interaction when working with LLMs. This paper provides the following contributions to research on prompt engineering that apply LLMs to automate software development tasks. First, it provides a framework for documenting patterns for structuring prompts to solve a range of problems so that they can be adapted to different domains. Second, it presents a catalog of patterns that have been applied successfully to improve the outputs of LLM conversations. Third, it explains how prompts can be built from multiple patterns and illustrates prompt patterns that benefit from combination with other prompt patterns.

Motivation & Objective

• Provide a framework for documenting prompt patterns to solve a range of LLM interaction problems.
• Introduce a catalog of domain-independent prompt patterns applied to automating software development tasks.
• Show how prompts can be built from multiple patterns and combined with other patterns to enhance outputs.

Proposed method

• Define a pattern-based framework that structures prompts similar to software design patterns.
• Classify and catalog 16 prompt patterns applicable to conversational LLMs and output generation.
• Describe pattern structure, intent, motivation, and sample implementations to enable reuse.

Experimental results

Research questions

• RQ1 How can prompt patterns be documented and transferred across domains?
• RQ2 What are essential prompt patterns for improving LLM output and interaction in software tasks?
• RQ3 How can multiple prompt patterns be composed to achieve broader goals?

Key findings

• Introduces a catalog of 16 prompt patterns categorized into five groups: Input Semantics, Output Customization, Error Identification, Prompt Improvement, and Interaction.
• Defines a Meta Language Creation pattern to allow alternative languages or notations for prompts.
• Presents Output Automater, Persona, Visualization Generator, Recipe, Template, Fact Check List, Reflection, Question Refinement, Alternative Approaches, Cognitive Verifier, Refusal Breaker, Flipped Interaction, Game Play, Infinite Generation, and Context Manager patterns.
• Describes fundamental contextual statements as a lightweight approach to documenting prompt ideas without rigid grammars.
• Shows that prompts can be composed from multiple patterns to meet complex interaction and output requirements.
• Notes evaluation using ChatGPT+ and emphasizes reuse, transferability, and domain adaptation of the patterns.