[Paper Review] OverPrompt: Enhancing ChatGPT through Efficient In-Context Learning

Jiazheng Li, Runcong Zhao+3 more|arXiv (Cornell University)|May 24, 2023

Artificial Intelligence in Healthcare and Education4 citations

TL;DR

OverPrompt proposes a cost-efficient zero-shot prompting strategy that batches multiple inputs into a single LLM inference call, reducing token and time costs while improving performance through in-context learning. By reusing task descriptions and leveraging LLMs' ability to generalize across grouped inputs, OverPrompt achieves lower inference costs and enhanced accuracy on text classification tasks, especially in sentiment analysis and fact-checking.

ABSTRACT

The remarkable performance of pre-trained large language models has revolutionised various natural language processing applications. Due to huge parametersizes and extensive running costs, companies or organisations tend to transfer the models to the target task by zero-shot prompting techniques. However, the prohibitive costs of tokens and time have hindered their adoption in applications. We propose OverPrompt, leveraging the in-context learning capability of LLMs to handle multiple task inputs, thereby reducing token and time costs. This approach could potentially improve task performance during API queries due to better conditional distribution mapping. Evaluated across diverse classification datasets, our experiments show that OverPrompt can achieve cost-efficient zero-shot classification without causing significant detriment to task performance, and in some cases, even improving it. An ablation study conducted on various LLMs, along with an investigation into the robustness of our prompting strategy to different input ordering, offers valuable insights into the broader applicability of our method across diverse tasks. These findings also suggest a more seamless integration of our method with LLMs through an API.

Motivation & Objective

• To reduce the high token and time costs associated with deploying large language models (LLMs) via API for zero-shot classification.
• To explore whether batching multiple unlabelled inputs in a single prompt can improve task performance through in-context learning.
• To design a prompting strategy that minimizes redundant task description repetition while maintaining or enhancing prediction accuracy.
• To evaluate the robustness of the method across different LLMs and input ordering configurations.
• To enable more efficient and scalable deployment of LLMs in production environments through cost-effective prompting.

Proposed method

• OverPrompt groups multiple unlabelled input instances into a single prompt, reusing the task description only once to reduce token consumption.
• The method leverages the in-context learning (ICL) capability of LLMs to improve conditional distribution mapping by providing contextual information across multiple inputs.
• A Bayesian inference framework is used to theoretically justify that incorporating multiple inputs improves distribution approximation and reduces format errors.
• The prompting template is designed to minimize redundancy, with task description stated once and multiple inputs formatted consistently to enhance model generalization.
• Output formatting is modified to reduce error rates and improve consistency, particularly in tasks with clear category boundaries.
• The approach is evaluated across ten diverse text classification datasets using GPT-3.5-turbo and GPT-4, with ablation studies on input ordering and model variants.

Figure 1: This illustration highlights the difference between traditional zero-shot classification prompting strategy and our OverPrompt. Deploying LLMs requires significant computational resources, so abandoning API queries is not practical. Our OverPrompt strategy prioritizes cost efficiency while

Experimental results

Research questions

• RQ1 Can batching multiple unlabelled inputs into a single prompt reduce token and time costs in zero-shot LLM inference?
• RQ2 Does providing contextual information from multiple inputs improve the model’s prediction accuracy in zero-shot classification?
• RQ3 How robust is the performance of OverPrompt to variations in input ordering and different LLM architectures?
• RQ4 In which types of classification tasks does OverPrompt provide performance gains, and why?
• RQ5 Can OverPrompt be effectively integrated into API-based LLM workflows without compromising model reliability?

Key findings

• OverPrompt significantly reduces both token and time costs by reusing the task description across multiple inputs in a single API call.
• The method improves classification performance on several datasets, particularly in sentiment analysis and fact-checking, where contextual comparison enhances decision-making.
• Performance gains are most pronounced with gpt-3.5-turbo and gpt-4 models, suggesting a strong alignment with their in-context learning mechanisms.
• The method is robust to input ordering, with minimal performance degradation across different sequences of inputs, indicating stability in real-world deployment.
• Ablation studies confirm that the strategy is effective even when input length is long, though performance may degrade with extremely long inputs due to context length limits.
• The approach reduces the carbon footprint of LLM inference by minimizing redundant API calls and token usage.

Figure 2: Illustration of ChatGPT struggling with similar sentences when input individually. Employing the OverPrompt strategy and cohesively grouping synthetic data from the "VitaminC" dataset may improve the performance of zero-shot inference.