[Paper Review] Large Language Models as Optimizers

Chengrun Yang, Xuezhi Wang+5 more|arXiv (Cornell University)|Sep 7, 2023

Topic Modeling90 citations

TL;DR

The paper introduces Optimization by PROMPTing (OPRO), using LLMs as optimizers to generate and evaluate new solutions in a prompt-driven loop, and demonstrates prompt optimization with LLMs across GSM8K and BBH, showing prompts optimized by LLMs can surpass human-designed prompts in some tasks.

ABSTRACT

Optimization is ubiquitous. While derivative-based algorithms have been powerful tools for various problems, the absence of gradient imposes challenges on many real-world applications. In this work, we propose Optimization by PROmpting (OPRO), a simple and effective approach to leverage large language models (LLMs) as optimizers, where the optimization task is described in natural language. In each optimization step, the LLM generates new solutions from the prompt that contains previously generated solutions with their values, then the new solutions are evaluated and added to the prompt for the next optimization step. We first showcase OPRO on linear regression and traveling salesman problems, then move on to our main application in prompt optimization, where the goal is to find instructions that maximize the task accuracy. With a variety of LLMs, we demonstrate that the best prompts optimized by OPRO outperform human-designed prompts by up to 8% on GSM8K, and by up to 50% on Big-Bench Hard tasks. Code at https://github.com/google-deepmind/opro.

Motivation & Objective

• Describe how LLMs can act as optimizers by iterating generations based on past solutions and their scores.
• Showcase case studies on continuous (linear regression) and discrete (TSP) optimization to illustrate capabilities and limitations.
• Demonstrate prompt optimization where LLMs optimize prompts to maximize task accuracy on NLP benchmarks.
• Evaluate multiple LLMs as optimizers and scorers across GSM8K and BBH to assess generality and transferability.

Proposed method

• Define Optimization by PROMPTing (OPRO) where an LLM, given a meta-prompt, generates new candidate solutions based on past solution-score history.
• In each step, evaluate new solutions and append their scores to the optimization trajectory within the meta-prompt.
• Incorporate an exploration-exploitation balance by tuning sampling temperature and generating multiple candidate solutions per step.
• Use a meta-prompt that contains optimization problem description, optimization trajectory, and exemplar problems to guide the optimizer.
• Apply OPRO to mathematical tasks (linear regression and TSP) to demonstrate black-box optimization capabilities.
• Apply OPRO to prompt optimization by using a scorer LLM to evaluate generated prompts and an optimizer LLM to generate new prompts, with a small training subset guiding the objective.
• Compare multiple LLMs as optimizers (text-bison, PaLM 2-L, PaLM 2-L-IT, gpt-3.5-turbo, gpt-4) and scorers across GSM8K and BBH.

(a) GSM8K

Experimental results

Research questions

• RQ1 Can LLMs serve as effective optimizers for both continuous and discrete problems using purely prompting-based interactions?
• RQ2 How well do LLMs optimize prompts to maximize task accuracy on NLP benchmarks like GSM8K and BBH?
• RQ3 What are the limitations and stability considerations when using LLMs as optimizers, especially for larger-scale problems?
• RQ4 How do different optimizer and scorer LLM combinations affect prompt optimization performance and transferability?
• RQ5 To what extent do optimized prompts transfer across related benchmarks (e.g., GSM8K to MultiArith and AQuA)?

Key findings

• OPRO enables LLMs to generate new solutions and improve optimization trajectories across tasks.
• Best prompts optimized by LLMs can surpass zero-shot human-designed prompts by up to 8% on GSM8K and by up to 50% on Big-Bench Hard tasks.
• GPT-4, text-bison, PaLM 2-L-IT, and PaLM 2-L show varying styles and strengths, with some achieving faster convergence and better optima.
• On TSP, GPT-4 significantly outperforms other LLMs in smaller problems, but performance degrades with larger n, where traditional heuristics can outperform LLMs in optimality gap.
• Prompt optimization demonstrates that small training subsets suffice to guide optimization and that optimized prompts transfer to related mathematics datasets.

(b) BBH movie_recommendation