[Paper Review] Diagnostic Questions: The NeurIPS 2020 Education Challenge.

Zichao Wang, Angus Lamb+10 more|arXiv (Cornell University)|Jul 23, 2020

Advanced Bandit Algorithms Research8 references28 citations

TL;DR

This paper presents the NeurIPS 2020 Education Challenge, which leverages over 20 million student answer records from the Eedi platform to develop models that predict student responses to diagnostic math questions, assess question quality, and generate personalized question sequences. The work enables automated, data-driven personalization of learning curricula by analyzing misconceptions revealed through multiple-choice responses.

ABSTRACT

Digital technologies are becoming increasingly prevalent in education, enabling personalized, high quality education resources to be accessible by students across the world. Importantly, among these resources are diagnostic questions: the answers that the students give to these questions reveal key information about the specific nature of misconceptions that the students may hold. Analyzing the massive quantities of data stemming from students' interactions with these diagnostic questions can help us more accurately understand the students' learning status and thus allow us to automate learning curriculum recommendations. In this competition, participants will focus on the students' answer records to these multiple-choice diagnostic questions, with the aim of 1) accurately predicting which answers the students provide; 2) accurately predicting which questions have high quality; and 3) determining a personalized sequence of questions for each student that best predicts the student's answers. These tasks closely mimic the goals of a real-world educational platform and are highly representative of the educational challenges faced today. We provide over 20 million examples of students' answers to mathematics questions from Eedi, a leading educational platform which thousands of students interact with daily around the globe. Participants to this competition have a chance to make a lasting, real-world impact on the quality of personalized education for millions of students across the world.

Motivation & Objective

• To predict student responses to multiple-choice diagnostic questions using large-scale interaction data.
• To evaluate the quality of diagnostic questions based on student response patterns.
• To generate personalized sequences of questions that best predict individual student answers.
• To support the development of automated, adaptive learning systems for personalized education at scale.
• To enable real-world impact on global access to high-quality, individualized math education.

Proposed method

• Leverages a dataset of over 20 million student answer records from Eedi, a global educational platform.
• Models student responses using machine learning techniques to infer misconceptions and knowledge states.
• Applies predictive modeling to forecast which answer choices students will select for each question.
• Uses response patterns to assess the diagnostic quality of questions, such as their ability to reveal specific misconceptions.
• Employs sequential modeling to generate optimal, personalized question ordering per student based on predicted response behavior.
• Integrates all components into a unified framework mimicking real-world educational platform workflows.

Experimental results

Research questions

• RQ1 How accurately can models predict student responses to diagnostic math questions using large-scale interaction data?
• RQ2 What features of student response patterns can be used to identify high-quality diagnostic questions?
• RQ3 How can personalized question sequences be generated to best predict individual student responses?
• RQ4 To what extent can automated systems replicate the diagnostic and adaptive capabilities of human tutors?
• RQ5 Can scalable machine learning models effectively support personalized learning at global scale?

Key findings

• The dataset of over 20 million student responses enables robust modeling of student misconceptions through answer patterns.
• Predictive models can accurately forecast student responses to multiple-choice diagnostic questions using interaction data.
• Question quality can be assessed based on consistency and diagnostic value revealed in student response distributions.
• Personalized question sequences can be generated to improve prediction accuracy of student answers.
• The framework demonstrates strong potential for real-world deployment in adaptive learning platforms.
• The competition framework supports scalable, data-driven personalization of math education for global learners.