[Paper Review] Can Large Language Model Comprehend Ancient Chinese? A Preliminary Test on ACLUE

Yixuan Zhang, Haonan Li|arXiv (Cornell University)|Oct 14, 2023

Topic ModelingComputer Science3 citations

TL;DR

This paper introduces ACLUE, the first comprehensive benchmark for evaluating large language models (LLMs) on ancient Chinese comprehension, comprising 15 tasks across lexical, syntactic, semantic, and knowledge-based understanding. In zero-shot evaluation, Chinese model ChatGLM2 achieved the highest average accuracy of 37.4%, significantly below modern Chinese benchmarks, highlighting substantial room for improvement in LLMs' grasp of ancient Chinese despite strong performance in modern language tasks.

ABSTRACT

Large language models (LLMs) have showcased remarkable capabilities in understanding and generating language. However, their ability in comprehending ancient languages, particularly ancient Chinese, remains largely unexplored. To bridge this gap, we present ACLUE, an evaluation benchmark designed to assess the capability of language models in comprehending ancient Chinese. ACLUE consists of 15 tasks cover a range of skills, spanning phonetic, lexical, syntactic, semantic, inference and knowledge. Through the evaluation of eight state-of-the-art LLMs, we observed a noticeable disparity in their performance between modern Chinese and ancient Chinese. Among the assessed models, ChatGLM2 demonstrates the most remarkable performance, achieving an average score of 37.4%. We have made our code and data public available.

Motivation & Objective

• To address the lack of standardized evaluation benchmarks for large language models (LLMs) in understanding ancient Chinese.
• To assess the zero-shot and in-context few-shot capabilities of state-of-the-art LLMs on a diverse set of ancient Chinese language tasks.
• To provide a unified, multiple-choice format benchmark compatible with zero-shot prompting and independent of task-specific fine-tuning.
• To analyze model performance across different task types, including lexical, syntactic, semantic, and knowledge-based comprehension.
• To evaluate the quality and difficulty of automatically generated versus manually curated questions in the benchmark.

Proposed method

• ACLUE constructs 15 tasks from a combination of manually curated questions from public sources and automatically generated questions from classical Chinese corpora.
• All tasks are formatted as multiple-choice questions with four options, ensuring consistency and compatibility with zero-shot and in-context learning evaluation protocols.
• Tasks span phonetic, lexical, syntactic, semantic, inference, and general knowledge skills, covering genres from poetry and prose to historical records and biographies across 2070 BCE to 1368 CE.
• Automatic question generation leverages existing annotated corpora, including word sense and named entity annotations, to create tasks like polysemy resolution and homographic character recognition.
• Model evaluation is conducted using both zero-shot and five-shot in-context prompting to assess generalization and few-shot adaptation without fine-tuning.
• Performance is measured via accuracy across all tasks, with statistical analysis comparing model performance across task categories and data origins (auto-generated vs. manually collected).

Figure 2: The performance of ChatGPT and ChatGLM2 on ACLUE of different categories.

Experimental results

Research questions

• RQ1 Can large language models comprehend ancient Chinese without task-specific fine-tuning?
• RQ2 How do state-of-the-art LLMs perform on a diverse set of ancient Chinese language understanding tasks?
• RQ3 What is the relative difficulty of auto-generated versus manually curated questions in evaluating ancient Chinese comprehension?
• RQ4 Which model architectures show superior generalization to ancient Chinese, and in which task categories do they excel?
• RQ5 What are the key challenges that limit LLMs' performance on ancient Chinese, particularly in tasks like homographic character resolution and long-context reading comprehension?

Key findings

• ChatGLM2 achieved the highest average accuracy of 37.4% across all 15 tasks in zero-shot evaluation, outperforming multilingual models like ChatGPT.
• ChatGPT achieved 36.9% average accuracy, showing slightly lower but more consistent performance across tasks compared to other models.
• All models performed poorly on homographic character resolution (T2), with accuracy near random guessing (25%), indicating a fundamental limitation in understanding phonetic substitutions in ancient Chinese.
• Reading comprehension (T8) posed a significant challenge due to long input lengths (average ~1,000 tokens), particularly affecting models like BLOOMZ, LLaMA, and Baichuan, suggesting limited long-context processing capability.
• BLOOMZ achieved 60.2% accuracy on couplet prediction (T5), likely due to training data overlap with the dataset source, highlighting data distribution sensitivity.
• Auto-generated questions in ACLUE demonstrated comparable difficulty and reliability to manually collected questions, supporting their use in large-scale LLM evaluation.

Figure 3: The performance comparison of LLMs on ACLUE across different data origins.