[Paper Review] PRompt Optimization in Multi-Step Tasks (PROMST): Integrating Human Feedback and Heuristic-based Sampling

Yongchao Chen, Jacob Arkin+4 more|arXiv (Cornell University)|Feb 13, 2024

Complex Systems and Decision Making4 citations

TL;DR

PROMST is a novel framework for automatic prompt optimization in multi-step LLM-driven tasks, combining human-designed feedback rules with a learned score prediction model to guide efficient candidate generation. It achieves 10.6%-29.3% relative performance gains over baselines across 11 multi-step tasks using diverse LLMs.

ABSTRACT

Prompt optimization aims to find the best prompt to a large language model (LLM) for a given task. LLMs have been successfully used to help find and improve prompt candidates for single-step tasks. However, realistic tasks for agents are multi-step and introduce new challenges: (1) Prompt content is likely to be more extensive and complex, making it more difficult for LLMs to analyze errors, (2) the impact of an individual step is difficult to evaluate, and (3) different people may have varied preferences about task execution. While humans struggle to optimize prompts, they are good at providing feedback about LLM outputs; we therefore introduce a new LLM-driven discrete prompt optimization framework PRompt Optimization in Multi-Step Tasks (PROMST) that incorporates human-designed feedback rules to automatically offer direct suggestions for improvement. We also use an extra learned heuristic model that predicts prompt performance to efficiently sample from prompt candidates. This approach significantly outperforms both human-engineered prompts and several other prompt optimization methods across 11 representative multi-step tasks (an average 10.6\%-29.3\% improvement to current best methods on five LLMs respectively). We believe our work can serve as a benchmark for automatic prompt optimization for LLM-driven multi-step tasks. Datasets and Codes are available at https://github.com/yongchao98/PROMST. Project Page is available at https://yongchao98.github.io/MIT-REALM-PROMST.

Motivation & Objective

• To address the challenge of automatic prompt optimization in complex, multi-step agent tasks where standard methods fail due to length and error complexity.
• To integrate human expertise into prompt optimization via pre-defined feedback rules that guide LLMs in generating improved prompts.
• To reduce costly evaluation overhead by training a score prediction model to heuristically select high-potential prompt candidates.
• To align optimized prompts with human preferences through rule-based evaluation criteria.
• To establish a benchmark for automatic prompt optimization in multi-step environments via open-sourced datasets and code.

Proposed method

• The framework uses a TaskLLM to execute actions in a multi-step environment, failing when preconditions are violated.
• Human-designed feedback rules automatically generate corrective feedback for each failure, which is then used as context for a PromptLLM to generate new prompt candidates.
• A score function evaluates each prompt’s performance on the environment, producing labeled prompt-score pairs for training.
• A learned score prediction model is fine-tuned online using these pairs to predict prompt quality and guide candidate sampling.
• The system iteratively evolves prompts through generations, using the score model to prioritize evaluation of high-potential candidates.
• Feedback rules are designed to enforce correctness, avoid loops, ensure goal alignment, and maintain action format consistency.

Experimental results

Research questions

• RQ1 Can human-designed feedback rules effectively guide LLMs in generating improved prompts for multi-step tasks?
• RQ2 How does integrating a learned score prediction model improve the efficiency and effectiveness of prompt optimization?
• RQ3 To what extent does PROMST outperform human-engineered and existing automatic prompt optimization methods in multi-step settings?
• RQ4 Can rule-based evaluation criteria help align optimized prompts with human preferences?
• RQ5 Is PROMST compatible and effective when combined with dynamic prompting strategies?

Key findings

• PROMST achieves 10.6% to 29.3% relative performance improvement over the best baseline methods across five LLMs on 11 multi-step tasks.
• The integration of human-designed feedback rules and a learned score model significantly outperforms both human-engineered prompts and existing automatic optimization techniques.
• PROMST demonstrates superior performance in multi-trial settings, especially when combined with dynamic prompting approaches.
• The human-designed evaluation rules effectively align optimized prompts with human preferences, improving task success and consistency.
• The framework is orthogonal to and compatible with established dynamic prompting strategies, enhancing their effectiveness.
• The authors release all datasets and code for 11 multi-step environments, establishing a benchmark for future research in automatic prompt optimization.