[Paper Review] How to Prompt? Opportunities and Challenges of Zero- and Few-Shot Learning for Human-AI Interaction in Creative Applications of Generative Models

D. D. Hai, Lukas Mecke+3 more|arXiv (Cornell University)|Sep 3, 2022

Educational Games and Gamification89 citations

TL;DR

The paper analyzes opportunities and challenges of prompting-based zero- and few-shot learning for interactive human-AI collaboration in creative tasks, and proposes four UI design goals to support prompting.

ABSTRACT

Deep generative models have the potential to fundamentally change the way we create high-fidelity digital content but are often hard to control. Prompting a generative model is a promising recent development that in principle enables end-users to creatively leverage zero-shot and few-shot learning to assign new tasks to an AI ad-hoc, simply by writing them down. However, for the majority of end-users writing effective prompts is currently largely a trial and error process. To address this, we discuss the key opportunities and challenges for interactive creative applications that use prompting as a new paradigm for Human-AI interaction. Based on our analysis, we propose four design goals for user interfaces that support prompting. We illustrate these with concrete UI design sketches, focusing on the use case of creative writing. The research community in HCI and AI can take these as starting points to develop adequate user interfaces for models capable of zero- and few-shot learning.

Motivation & Objective

• Motivate prompting as a new paradigm for Human-AI interaction in creative tasks using generative models.
• Identify key opportunities and challenges for end-user prompting in creativity-focused applications.
• Propose design goals and illustrate UI concepts to support zero- and few-shot prompting for end users.

Proposed method

• Conducted two one-hour brainstorming sessions with five HCI researchers to collect opportunities and challenges of prompting for creative interaction.
• Performed a literature review on interactive prompting and prompt engineering.
• Synthesized findings into four design goals for UIs supporting prompting and illustrated with UI sketches focusing on creative writing.

Experimental results

Research questions

• RQ1 What opportunities do zero- and few-shot prompting offer for end-user creative tasks?
• RQ2 What are the main challenges in designing interfaces to support prompting for human-AI collaboration in creative domains?
• RQ3 What concrete UI design directions can enable effective prompt formulation, combination, application, and representation?

Key findings

• Prompting enables end-user programming of creative tools by defining ad hoc AI tools through natural language.
• Prompts can extend and augment creative expressiveness across modalities and styles.
• Prompts can provide inspiration and feedback to overcome creative blocks and support iterative drafting.
• Users need interfaces that support prompt formulation, combination, application, and visualization of prompts.
• There are notable challenges including trial-and-error usage, prompt IO representation, computational costs, generalization across models, and ethical concerns.