[Paper Review] Responsible AI for Test Equity and Quality: The Duolingo English Test as a Case Study

Jill Burstein, Geoffrey T. LaFlair+3 more|arXiv (Cornell University)|Aug 28, 2024

Explainable Artificial Intelligence (XAI)6 citations

TL;DR

This paper presents a case study of the Duolingo English Test (DET) and its Responsible AI (RAI) practices aimed at ensuring test equity and quality.

ABSTRACT

Artificial intelligence (AI) creates opportunities for assessments, such as efficiencies for item generation and scoring of spoken and written responses. At the same time, it poses risks (such as bias in AI-generated item content). Responsible AI (RAI) practices aim to mitigate risks associated with AI. This chapter addresses the critical role of RAI practices in achieving test quality (appropriateness of test score inferences), and test equity (fairness to all test takers). To illustrate, the chapter presents a case study using the Duolingo English Test (DET), an AI-powered, high-stakes English language assessment. The chapter discusses the DET RAI standards, their development and their relationship to domain-agnostic RAI principles. Further, it provides examples of specific RAI practices, showing how these practices meaningfully address the ethical principles of validity and reliability, fairness, privacy and security, and transparency and accountability standards to ensure test equity and quality.

Motivation & Objective

• Motivate the need for Responsible AI in high-stakes assessments to safeguard test quality and equity.
• Describe the development of DET’s RAI standards and their relationship to domain-agnostic RAI principles.
• Illustrate how RAI practices address key ethical principles to support valid inferences and fair outcomes.
• Show how DET’s RAI framework contributes to transparency, accountability, privacy, and security in assessment.

Proposed method

• Present DET’s RAI standards and explain their development.
• Relate DET RAI standards to broader, domain-agnostic RAI principles.
• Provide concrete examples of RAI practices applied in DET.
• Discuss how the practices support validity, reliability, fairness, privacy and security, and transparency and accountability.

Experimental results

Research questions

• RQ1 How do DET’s Responsible AI standards align with broader domain-agnostic RAI principles?
• RQ2 What specific RAI practices are employed in DET to address validity, reliability, fairness, privacy and security, and transparency and accountability?
• RQ3 In what ways do these RAI practices contribute to test equity and test quality in a high-stakes AI-powered English language assessment?

Key findings

• The chapter outlines DET’s RAI standards and their development process.
• It provides examples of specific RAI practices implemented in DET.
• The discussion connects these practices to ethical principles of validity, reliability, fairness, privacy and security, and transparency and accountability.
• The findings illustrate how RAI practices can meaningfully address test equity and quality in an AI-driven assessment.