[Paper Review] AI Agents That Matter

Sayash Kapoor, Benedikt Stroebl+3 more|arXiv (Cornell University)|Jul 1, 2024

Computability, Logic, AI Algorithms4 citations

TL;DR

This paper critiques current AI agent benchmarks for overemphasizing accuracy at the expense of cost, reproducibility, and real-world applicability. It proposes cost-controlled evaluation, joint optimization of accuracy and cost via Pareto curves, distinct benchmarking for model vs. downstream developers, principled holdout sets to prevent overfitting, and standardized evaluation practices—demonstrating that simpler, cheaper agents can match or exceed SOTA performance on HumanEval and HotPotQA.

ABSTRACT

AI agents are an exciting new research direction, and agent development is driven by benchmarks. Our analysis of current agent benchmarks and evaluation practices reveals several shortcomings that hinder their usefulness in real-world applications. First, there is a narrow focus on accuracy without attention to other metrics. As a result, SOTA agents are needlessly complex and costly, and the community has reached mistaken conclusions about the sources of accuracy gains. Our focus on cost in addition to accuracy motivates the new goal of jointly optimizing the two metrics. We design and implement one such optimization, showing its potential to greatly reduce cost while maintaining accuracy. Second, the benchmarking needs of model and downstream developers have been conflated, making it hard to identify which agent would be best suited for a particular application. Third, many agent benchmarks have inadequate holdout sets, and sometimes none at all. This has led to agents that are fragile because they take shortcuts and overfit to the benchmark in various ways. We prescribe a principled framework for avoiding overfitting. Finally, there is a lack of standardization in evaluation practices, leading to a pervasive lack of reproducibility. We hope that the steps we introduce for addressing these shortcomings will spur the development of agents that are useful in the real world and not just accurate on benchmarks.

Motivation & Objective

• Address the overreliance on accuracy as the sole metric in AI agent evaluation, which leads to unnecessarily complex and expensive agents.
• Identify that model developers and downstream application developers have divergent benchmarking needs, leading to misalignment in evaluation goals.
• Expose how inadequate holdout sets in benchmarks enable shortcut learning and overfitting, undermining generalization.
• Highlight the lack of standardization and reproducibility in agent evaluations, inflating reported performance and reducing trust in results.
• Promote the development of AI agents that are not only accurate but also cost-effective, generalizable, and practically useful in real-world applications.

Proposed method

• Introduce three simple baseline agents (e.g., chain-of-thought, self-consistency) to demonstrate that high accuracy can be achieved with significantly lower inference cost than complex SOTA agents.
• Visualize evaluation results as a Pareto frontier of accuracy vs. cost, enabling joint optimization of both metrics using modified DSPy framework.
• Design and implement a framework for generating holdout sets at multiple levels of generality (e.g., task, domain, instruction style) to detect overfitting to benchmark patterns.
• Conduct case studies on HumanEval, HotPotQA, NovelQA, and WebArena to expose reproducibility issues and overfitting risks in existing benchmarks.
• Release open-source code and a web application to reproduce results, visualize cost-accuracy tradeoffs, and support community adoption of standardized evaluation practices.
• Apply cost-aware evaluation by measuring actual inference cost (in USD) rather than relying on proxies like model parameters or token counts.

Experimental results

Research questions

• RQ1 To what extent can simple baseline agents achieve SOTA performance on HumanEval while incurring significantly lower cost than complex agent architectures?
• RQ2 How can joint optimization of accuracy and cost be formalized and implemented in agent development, and what are the resulting trade-offs?
• RQ3 Why do existing benchmarks fail to distinguish between model development and downstream application needs, and how can this be corrected?
• RQ4 What types of overfitting are possible in agent benchmarks, and how can holdout sets at different levels of generality prevent them?
• RQ5 To what extent are current agent evaluations reproducible, and what systemic issues undermine the reliability of reported performance metrics?

Key findings

• Simple baseline agents (e.g., chain-of-thought, self-consistency) outperform multiple SOTA complex agent architectures on HumanEval while reducing inference cost by orders of magnitude.
• Joint optimization of accuracy and cost via Pareto frontier analysis in the DSPy framework successfully reduces cost by up to 70% on HotPotQA while maintaining SOTA accuracy.
• Benchmark evaluations for model development often mislead downstream developers because they rely on proxies like model size rather than actual dollar costs of inference.
• Overfitting to benchmarks is pervasive—case studies on WebArena and HumanEval reveal reproducibility issues, including inconsistent data splits and unreported hyperparameters, inflating accuracy estimates.
• The lack of standardized evaluation practices leads to inflated performance claims; for example, error bars were missing or inconsistent in published evaluations, undermining confidence in results.
• The authors release a fully reproducible codebase, web application for cost-accuracy tradeoff exploration, and a joint optimizer to support community-wide adoption of more rigorous evaluation standards.