[Paper Review] AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models

Wanjun Zhong, Ruixiang Cui+7 more|arXiv (Cornell University)|Apr 13, 2023

Artificial Intelligence in Healthcare and Education61 citations

TL;DR

AGIEval presents a bilingual, human-exams-based benchmark with 8,062 questions across 20 tasks to evaluate foundation models, revealing GPT-4 excels on some human-centric tests but struggles with complex reasoning and domain-specific knowledge.

ABSTRACT

Evaluating the general abilities of foundation models to tackle human-level tasks is a vital aspect of their development and application in the pursuit of Artificial General Intelligence (AGI). Traditional benchmarks, which rely on artificial datasets, may not accurately represent human-level capabilities. In this paper, we introduce AGIEval, a novel benchmark specifically designed to assess foundation model in the context of human-centric standardized exams, such as college entrance exams, law school admission tests, math competitions, and lawyer qualification tests. We evaluate several state-of-the-art foundation models, including GPT-4, ChatGPT, and Text-Davinci-003, using this benchmark. Impressively, GPT-4 surpasses average human performance on SAT, LSAT, and math competitions, attaining a 95% accuracy rate on the SAT Math test and a 92.5% accuracy on the English test of the Chinese national college entrance exam. This demonstrates the extraordinary performance of contemporary foundation models. In contrast, we also find that GPT-4 is less proficient in tasks that require complex reasoning or specific domain knowledge. Our comprehensive analyses of model capabilities (understanding, knowledge, reasoning, and calculation) reveal these models' strengths and limitations, providing valuable insights into future directions for enhancing their general capabilities. By concentrating on tasks pertinent to human cognition and decision-making, our benchmark delivers a more meaningful and robust evaluation of foundation models' performance in real-world scenarios. The data, code, and all model outputs are released in https://github.com/ruixiangcui/AGIEval.

Motivation & Objective

• Center evaluation on tasks aligned with human cognition and decision-making via official exams.
• Provide a robust, standardized, automatic metric using objective question formats.
• Benchmark multilingual capabilities with English and Chinese tasks.
• Release model outputs to foster transparency and reproducibility in LM evaluation.

Proposed method

• Compile questions from high-standard, official exams (Gaokao, SAT, LSAT, GMAT, AMC/AIME, civil service, lawyer exams).
• Include only objective items (multiple-choice and fill-in-the-blank) for standardized scoring.
• Use accuracy for multiple-choice and Exact Match/F1 for fill-in-the-blank questions as metrics.
• Evaluate models under zero-shot and few-shot settings with and without Chain-of-Thought prompting.
• Use Azure OpenAI Service APIs for Text-Davinci-003, ChatGPT, and GPT-4, with fixed generation settings (temperature 0, max tokens 2048).
• Release all model outputs to support analysis and reproducibility.

Experimental results

Research questions

• RQ1 How well do state-of-the-art foundation models perform on human-level, real-world tasks derived from official exams?
• RQ2 What are the strengths and limitations of these models in understanding, knowledge, reasoning, and calculation across bilingual tasks?
• RQ3 Do chain-of-thought prompts and few-shot scenarios improve performance on human-centric reasoning tasks?
• RQ4 How does model performance compare to average and top human test-takers on various exams?

Key findings

• GPT-4 surpasses average human performance on SAT, LSAT, and math competitions under zero-shot CoT settings.
• GPT-4 achieves 95% accuracy on SAT Math and 92.5% accuracy on the English test of the Chinese Gaokao.
• Models struggle with tasks requiring complex reasoning or specific domain knowledge (e.g., law, chemistry, physics).
• Evaluation across understanding, knowledge, reasoning, and calculation reveals distinct strengths and limitations of each model.
• The benchmark provides insights into directions for improving general capabilities toward AGI.