[Paper Review] Towards Optimizing the Costs of LLM Usage

Shivanshu Shekhar, Tanishq Dubey+4 more|arXiv (Cornell University)|Jan 29, 2024

Scheduling and Optimization Algorithms4 citations

TL;DR

This paper proposes QC-Opt, a framework that reduces LLM usage costs by estimating output quality without invoking models and optimizing LLM selection and input token length. It achieves 40–90% cost reduction with 4–7% quality improvement via quality-aware LLM routing, token optimization heuristics, and a knapsack-based heuristic application strategy.

ABSTRACT

Generative AI and LLMs in particular are heavily used nowadays for various document processing tasks such as question answering and summarization. However, different LLMs come with different capabilities for different tasks as well as with different costs, tokenization, and latency. In fact, enterprises are already incurring huge costs of operating or using LLMs for their respective use cases. In this work, we propose optimizing the usage costs of LLMs by estimating their output quality (without actually invoking the LLMs), and then solving an optimization routine for the LLM selection to either keep costs under a budget, or minimize the costs, in a quality and latency aware manner. We propose a model to predict the output quality of LLMs on document processing tasks like summarization, followed by an LP rounding algorithm to optimize the selection of LLMs. We study optimization problems trading off the quality and costs, both theoretically and empirically. We further propose a sentence simplification model for reducing the number of tokens in a controlled manner. Additionally, we propose several deterministic heuristics for reducing tokens in a quality aware manner, and study the related optimization problem of applying the heuristics optimizing the quality and cost trade-off. We perform extensive empirical validation of our methods on not only enterprise datasets but also on open-source datasets, annotated by us, and show that we perform much better compared to closest baselines. Our methods reduce costs by 40%- 90% while improving quality by 4%-7%. We will release the annotated open source datasets to the community for further research and exploration.

Motivation & Objective

• To address the high and variable costs of using LLMs for document processing tasks like summarization and question answering.
• To estimate LLM output quality without invoking the models, enabling cost-effective routing decisions.
• To optimize both LLM selection and input token count to minimize costs while maintaining or improving quality and latency.
• To develop a token reduction pipeline using heuristics that control quality degradation and maximize token savings.
• To validate the framework empirically on enterprise and open-source datasets, demonstrating cost savings and quality gains.

Proposed method

• Proposes a quality estimation model using a custom loss function to predict LLM output quality for summarization tasks without running the models.
• Introduces an LP rounding algorithm to solve the LLM selection problem under cost, quality, and latency constraints.
• Designs a token optimization module using 8 deterministic heuristics (e.g., stop word removal, acronym simplification) to reduce input length.
• Models the optimal application of heuristics per sentence as a knapsack problem with quality loss as cost and token savings as profit.
• Uses brute force over small heuristic sets to determine the optimal order of operations per sentence.
• Employs a smart router that selects the best LLM and applies token optimizations based on predicted quality and cost constraints.

Experimental results

Research questions

• RQ1 Can LLM output quality be accurately predicted without invoking the model, enabling cost-effective routing?
• RQ2 How can LLM selection and input token length be jointly optimized to minimize costs while maintaining quality?
• RQ3 What is the trade-off between token reduction and quality degradation when applying heuristic-based compression?
• RQ4 Can a knapsack-based formulation effectively guide the optimal application of token reduction heuristics?
• RQ5 How do the proposed methods compare to baselines in terms of cost, quality, and latency across diverse document processing tasks?

Key findings

• The proposed QC-Opt framework reduces LLM usage costs by 40–90% on enterprise and open datasets while improving output quality by 4–7%.
• The quality estimation model enables accurate prediction of LLM performance without actual inference, reducing runtime costs and latency.
• Token optimization via heuristics achieves up to 22.07% compression on summarization datasets with controlled quality loss.
• The knapsack-based heuristic application strategy reduces quality degradation by 30–50% compared to brute-force heuristic application.
• Latency is reduced by approximately 13% due to routing more queries to faster, cheaper models like Text-Curie.
• User studies confirm that human perception of output quality aligns with automated metrics like BertScore, validating the model’s predictive accuracy.