[Paper Review] PromptAid: Prompt Exploration, Perturbation, Testing and Iteration using Visual Analytics for Large Language Models

Aditi Mishra, Utkarsh Soni+4 more|arXiv (Cornell University)|Apr 4, 2023

Explainable Artificial Intelligence (XAI)21 citations

TL;DR

PromptAid is a visual-analytics system that helps non-experts craft, perturb, test, and iterate prompts for LLMs using keyword perturbations, paraphrasing, and in-context examples, with global and instance-level analysis.

ABSTRACT

Large Language Models (LLMs) have gained widespread popularity due to their ability to perform ad-hoc Natural Language Processing (NLP) tasks with a simple natural language prompt. Part of the appeal for LLMs is their approachability to the general public, including individuals with no prior technical experience in NLP techniques. However, natural language prompts can vary significantly in terms of their linguistic structure, context, and other semantics. Modifying one or more of these aspects can result in significant differences in task performance. Non-expert users may find it challenging to identify the changes needed to improve a prompt, especially when they lack domain-specific knowledge and lack appropriate feedback. To address this challenge, we present PromptAid, a visual analytics system designed to interactively create, refine, and test prompts through exploration, perturbation, testing, and iteration. PromptAid uses multiple, coordinated visualizations which allow users to improve prompts by using the three strategies: keyword perturbations, paraphrasing perturbations, and obtaining the best set of in-context few-shot examples. PromptAid was designed through an iterative prototyping process involving NLP experts and was evaluated through quantitative and qualitative assessments for LLMs. Our findings indicate that PromptAid helps users to iterate over prompt template alterations with less cognitive overhead, generate diverse prompts with help of recommendations, and analyze the performance of the generated prompts while surpassing existing state-of-the-art prompting interfaces in performance.

Motivation & Objective

• Address the challenge that non-experts struggle to craft optimal prompts for LLMs.
• Provide a visual analytics interface to explore the prompt space and analyze performance at global and instance levels.
• Offer semi-automated recommendations for linguistic and contextual perturbations (keywords, paraphrases, k-shot examples).
• Support iterative prompt refinement and testing on in-distribution and out-of-distribution data.

Proposed method

• Develop PromptAid with six linked panels that support model/domain selection, prompt exploration, instance-level analysis, version comparison, perturbation recommendations, and testing.
• Embed prompts in a latent embedding space and cluster them by similarity for exploration.
• Compute perturbation recommendations using KD-Trees for keywords, Parrot for paraphrases, and KNN for k-shot examples.
• Use heuristic accuracy on a test set to drive perturbation sensitivities for next-step guidance.
• Incorporate testing on in-distribution and out-of-distribution data points to evaluate prompts.
• Validate the system through case studies and a within-subject user study with non-experts.

Experimental results

Research questions

• RQ1 How can non-experts effectively explore and optimize prompt templates for LLMs with visual analytics?
• RQ2 What perturbation strategies (linguistic and contextual) most improve prompt performance in zero-shot and few-shot settings?
• RQ3 Can interactive visualization reduce cognitive load while guiding prompt Iteration and testing?
• RQ4 Do semi-automatic recommendations improve prompt performance compared to baseline prompting interfaces?

Key findings

• PromptAid enables iterative prompt template alterations with reduced cognitive load.
• The system generates diverse prompts aided by recommendations and analyzes their performance effectively.
• PromptAid outperforms existing prompting interfaces in enabling higher performance on tested tasks.
• Users utilizing PromptAid reported high usefulness of interactive visualizations and perturbation recommendations.