[论文解读] Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence

Timothy R. McIntosh, Teo Sušnjak+4 more|arXiv (Cornell University)|Feb 15, 2024

Topic Modeling被引用 42

一句话总结

本文对23个最先进的LLM基准进行了批判性评估，识别了在技术、流程和人员方面的主要不足之处，并提出一个统一框架以及行为审计，以改进未来的评估。

ABSTRACT

The rapid rise in popularity of Large Language Models (LLMs) with emerging capabilities has spurred public curiosity to evaluate and compare different LLMs, leading many researchers to propose their own LLM benchmarks. Noticing preliminary inadequacies in those benchmarks, we embarked on a study to critically assess 23 state-of-the-art LLM benchmarks, using our novel unified evaluation framework through the lenses of people, process, and technology, under the pillars of benchmark functionality and integrity. Our research uncovered significant limitations, including biases, difficulties in measuring genuine reasoning, adaptability, implementation inconsistencies, prompt engineering complexity, evaluator diversity, and the overlooking of cultural and ideological norms in one comprehensive assessment. Our discussions emphasized the urgent need for standardized methodologies, regulatory certainties, and ethical guidelines in light of Artificial Intelligence (AI) advancements, including advocating for an evolution from static benchmarks to dynamic behavioral profiling to accurately capture LLMs' complex behaviors and potential risks. Our study highlighted the necessity for a paradigm shift in LLM evaluation methodologies, underlining the importance of collaborative efforts for the development of universally accepted benchmarks and the enhancement of AI systems' integration into society.

研究动机与目标

• Identify common inadequacies in state-of-the-art LLM benchmarks across technological, processual, and human dimensions.
• Propose a unified evaluation framework aligned with cybersecurity principles focusing on functionality and security.
• Analyze 23 benchmarks to assess prevalence of inadequacies and gaps in real-world applicability and safety.
• Suggest extending benchmarks with LLM behavioral profiling and audits to enhance inclusivity and security insights.

提出的方法

• Develop a unified evaluation framework for LLM benchmarks that integrates people, process, and technology.
• Apply a reverse-thinking counter-example approach to identify inadequacies and categorize them as present-but-unacknowledged, acknowledged-but-unresolved, or addressed.
• Conduct a structured manual evaluation across technological, processual, and human dimensions (Appendices A–C) to assess benchmarks.
• Systematically analyze 23 benchmarks to map inadequacies and their prevalence (Table II reference) and discuss implications for methodology.
• Highlight the need for dynamic, behavior-based benchmarking and regulatory/ethical guidelines.

实验结果

研究问题

• RQ1 How can one identify, categorize, and explain the common inadequacies of state-of-the-art LLM benchmarks?
• RQ2 Do the identified inadequacies manifest in popular benchmarks, and to what extent are they present or acknowledged?
• RQ3 What should a comprehensive LLM benchmark evaluation include, considering functionality and security for societal impact?

主要发现

• Benchmarks exhibit biases, inconsistencies, and gaps in evaluating genuine reasoning versus technical optimization.
• There is a persistent tension between model helpfulness and harmlessness in evaluations, especially in open-ended contexts.
• Linguistic variability and embedded logics across languages are often ignored, privileging English or Simplified Chinese with limited multilingual grounding.
• Evaluations frequently overlook cybersecurity aspects, such as adversarial or ideological manipulation risks.
• A unified people-process-technology framework is proposed to guide more holistic and secure LLM benchmarking.
• Behavioral profiling and audits of LLMs are proposed as extensions to current benchmarks to improve inclusivity and safety insights.