[Paper Review] C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models

Yuzhen Huang, Yuzhuo Bai+11 more|arXiv (Cornell University)|May 15, 2023

Topic Modeling90 citations

TL;DR

C-Eval is a comprehensive Chinese evaluation suite with 13,948 multiple-choice questions across 52 disciplines and four difficulty levels, plus C-Eval Hard for advanced reasoning; GPT-4 leads performance among tested models.

ABSTRACT

New NLP benchmarks are urgently needed to align with the rapid development of large language models (LLMs). We present C-Eval, the first comprehensive Chinese evaluation suite designed to assess advanced knowledge and reasoning abilities of foundation models in a Chinese context. C-Eval comprises multiple-choice questions across four difficulty levels: middle school, high school, college, and professional. The questions span 52 diverse disciplines, ranging from humanities to science and engineering. C-Eval is accompanied by C-Eval Hard, a subset of very challenging subjects in C-Eval that requires advanced reasoning abilities to solve. We conduct a comprehensive evaluation of the most advanced LLMs on C-Eval, including both English- and Chinese-oriented models. Results indicate that only GPT-4 could achieve an average accuracy of over 60%, suggesting that there is still significant room for improvement for current LLMs. We anticipate C-Eval will help analyze important strengths and shortcomings of foundation models, and foster their development and growth for Chinese users.

Motivation & Objective

• Assess advanced knowledge and reasoning abilities of foundation models in a Chinese context across multiple disciplines.
• Provide a multi-level (middle school to professional) evaluation with four domains: STEM, humanities, social science, and other.
• Mitigate data leakage by sourcing from mock/local exams and processing questions into a standardized four-option format.
• Enable analysis of model strengths/weaknesses and support development of Chinese-language LLMs.

Proposed method

• Assemble 13,948 four-option multiple-choice questions from mock exams and local Chinese exams across 52 subjects.
• Standardize each item into four choices with LaTeX formatting for STEM content where needed.
• Split data into development, validation, and test sets per subject; provide few-shot explanations for development items.
• Evaluate models in zero-shot and few-shot (five-shot) answer-only settings, plus chain-of-thought settings for some models.
• Maintain a private test set to prevent data leakage, with public leaderboard for submissions.

Experimental results

Research questions

• RQ1 How do state-of-the-art LLMs perform on a broad, Chinese-language exam-style benchmark across many disciplines?
• RQ2 What is the gap between Chinese-oriented and English-oriented models on Chinese knowledge and reasoning tasks?
• RQ3 How does few-shot prompting and chain-of-thought reasoning affect performance on complex Chinese exam questions?
• RQ4 What are the strengths and limitations of current LLMs in advanced Chinese reasoning tasks (C-Eval Hard)?

Key findings

• GPT-4 achieves the highest average accuracy, exceeding 60% on average across zero-shot and few-shot settings.
• Among Chinese-oriented models, GLM-130B is the strongest, but still trails ChatGPT in overall average by a notable margin, especially in STEM tasks.
• Few-shot prompting generally helps some models but can hurt others that were instruction-tuned without few-shot alignment.
• Chain-of-thought prompting does not universally improve performance and can reduce accuracy for several models, though it helps some (e.g., GPT-4 in certain subjects).
• On C-Eval Hard, GPT-4, ChatGPT, and Claude show meaningful gains with chain-of-thought, yet overall accuracy remains low for most models, highlighting the challenge of advanced reasoning.