[Paper Review] Beyond Accuracy: Towards a Robust Evaluation Methodology for AI Systems for Language Education

Sister James Edgell, Wm. Matthew Kennedy+4 more|arXiv (Cornell University)|Mar 20, 2026

Artificial Intelligence in Healthcare and Education0 citations

TL;DR

Introduces L2-Bench, a holistic, taxonomy-guided benchmark to evaluate AI in second-language education through learning experience design, with a pilot validation and a plan for practitioner validation across 1,000+ tasks.

ABSTRACT

The rapid adoption of large language models in AI-powered language education has created an urgent need for evaluations that assess pedagogical effectiveness, particularly in language learning--one of the most common LLM use cases (Tamkin et al. 2024, Costa-Gomes et al. 2025). With only narrowly defined task-specific evaluations of AI system capabilities in second language (L2) education existing in the literature, we require more holistic approaches in this AI for education space. To address this gap, we introduce L2-Bench, a novel evaluation benchmark grounded in a validated "language learning experience designer" construct to assess AI capabilities across L2 education contexts. Our methodology integrates pedagogical theory, sociotechnical AI evaluation methods, and operationalizes a hierarchical taxonomy to structure an expert-curated dataset of over 1,000 authentic rubric-scored task-response pairs with measurement and scoring pipeline. We report the results of a pilot validation exercise (N = 39) on an initial sample of our dataset (tasks were validated as authentic [M = 4.23 out of 5], but criteria scores were lower [M = 3.94], with universally poor inter-annotator agreement despite good internal consistency), alongside the experimental design for our follow-up practitioner data validation study as we iterate and scale to the full dataset. Ultimately, this research not only offers methodological lessons towards a more context-specific AI evaluations ecosystem, but also works towards better design of reproducible evaluations for AI systems deployed to educational contexts.

Motivation & Objective

• Define a hierarchical competency taxonomy for learning experience design in L2 education.
• Operationalize a large, authentic dataset of task-response pairs aligned to the taxonomy.
• Develop a rubric-based scoring pipeline with automated scoring and open-response evaluation.
• Pilot validate taxonomy, measures, and initial dataset to guide future scaling and validation.

Proposed method

• Develop a two-level, 12-competency taxonomy with 30 sub-competencies for learning experience design in L2 education.
• Create over 1,000 authentic task-response pairs using a hybrid human-AI authoring workflow from design to publish.
• Employ binary, rubric-based scoring with consensus, task-specific, and universal criteria and a scoring pipeline using an auto-scorer.
• Use system prompts to elicit task responses and reference answers, enabling standardized scoring while minimizing leakage.
• Pilot validate the taxonomy and dataset with 39 participants and 325 tasks to assess authenticity and criteria quality.
• Plan practitioner data validation across diverse stakeholder groups to measure authenticity, criteria adequacy, and auto-scorer validity.

Experimental results

Research questions

• RQ1 How can a hierarchical taxonomy of L2 learning experience design competencies be defined and validated?
• RQ2 Can an open-response task dataset aligned to the taxonomy provide reliable, rubric-based AI evaluation in L2 education?
• RQ3 What is the feasibility and design of a scalable auto-scorer and validation workflow for L2-Bench?
• RQ4 How will practitioner validation inform and refine the benchmark before full release?

Key findings

• Pilot task authenticity was rated high (M=4.23/5) but criteria quality was lower (M=3.94/5) with universally poor inter-annotator agreement.
• Inter-annotator agreement (IAA) for criteria scores was universally poor across competencies (Krippendorff’s Alpha up to -0.01; median negative values).
• Internal item consistency (Cronbach’s Alpha) was high (IIC overall=0.95) despite low IAA, suggesting evaluators applied different standards.
• A 12-competency, 30-sub-competency taxonomy covers course planning through professional development as learning experience design in L2 education.
• A hybrid design-draft-review-approval-publish workflow supports scalable task generation with rigorous pedagogical grounding.