[Paper Review] Results and Insights from Diagnostic Questions: The NeurIPS 2020 Education Challenge

Zichao Wang, Angus Lamb+10 more|arXiv (Cornell University)|Apr 8, 2021

Glioma Diagnosis and Treatment4 citations

TL;DR

This paper presents the NeurIPS 2020 Education Challenge, introducing a large-scale, real-world dataset of over 400 million student answers to multiple-choice diagnostic questions. It proposes four data mining tasks—answer prediction, matrix completion, question quality assessment, and personalized question selection—using unsupervised and active learning methods to enable scalable, data-driven personalization in online education.

ABSTRACT

This competition concerns educational diagnostic questions, which are pedagogically effective, multiple-choice questions (MCQs) whose distractors embody misconceptions. With a large and ever-increasing number of such questions, it becomes overwhelming for teachers to know which questions are the best ones to use for their students. We thus seek to answer the following question: how can we use data on hundreds of millions of answers to MCQs to drive automatic personalized learning in large-scale learning scenarios where manual personalization is infeasible? Success in using MCQ data at scale helps build more intelligent, personalized learning platforms that ultimately improve the quality of education en masse. To this end, we introduce a new, large-scale, real-world dataset and formulate 4 data mining tasks on MCQs that mimic real learning scenarios and target various aspects of the above question in a competition setting at NeurIPS 2020. We report on our NeurIPS competition in which nearly 400 teams submitted approximately 4000 submissions, with encouragingly diverse and effective approaches to each of our tasks.

Motivation & Objective

• Address the challenge of scalable, personalized education by leveraging real-world student response data to automate question selection and improve learning outcomes.
• Overcome the limitations of manual personalization in large-scale online learning platforms by using machine learning to adapt content to individual student needs.
• Develop and evaluate novel machine learning methods for educational data mining, particularly in sparse, categorical, and dynamic learning environments.
• Advance the state of the art in AI for education by providing a rich, publicly available dataset with extensive metadata on students, questions, and answer patterns.
• Explore the integration of metadata (e.g., skills, demographics, timestamps) and multimodal data (e.g., question images) to improve modeling performance and educational insights.

Proposed method

• Curated a large-scale, real-world dataset of 400+ million student answers to diagnostic multiple-choice questions from the Eedi platform, with rich metadata including skills, timestamps, and question images.
• Formulated four distinct tasks: (1) answer prediction using matrix completion on sparse student-question-response data; (2) matrix completion with categorical, unordered data; (3) automatic question quality assessment using response patterns; and (4) personalized question selection via active learning and ranking.
• Applied unsupervised and self-supervised learning techniques to handle the sparsity and dynamic nature of student learning, particularly in low-data regimes.
• Integrated student and question metadata (e.g., demographics, skill tags, timestamps) to improve prediction and ranking performance across tasks.
• Used multi-modal features from question images and text to support cross-modal learning and improve diagnostic question modeling.
• Evaluated solutions using real-world performance metrics, including prediction accuracy, AUC, and ranking quality, across diverse, non-uniform data distributions.

Experimental results

Research questions

• RQ1 How can we accurately predict student responses to diagnostic multiple-choice questions using sparse, real-world data?
• RQ2 What techniques enable effective matrix completion for unordered, categorical response data in educational settings?
• RQ3 How can we automatically assess the quality of diagnostic questions based on student response patterns and misconceptions?
• RQ4 What strategies enable personalized, adaptive question selection that maximizes learning efficiency and prediction accuracy?
• RQ5 How can metadata (e.g., skills, demographics, timestamps) and multi-modal data (e.g., images, text) be leveraged to improve educational modeling and personalization?

Key findings

• Nearly 400 teams participated in the NeurIPS 2020 competition, submitting approximately 4,000 solutions, demonstrating strong global interest in AI-driven educational personalization.
• Leading solutions for answer prediction achieved high AUC scores, indicating strong performance in modeling student knowledge states from sparse response data.
• The dataset enabled novel approaches to question quality assessment, with top models identifying high-quality diagnostic questions based on response pattern consistency and misconception alignment.
• Personalized question selection models outperformed random and heuristic baselines, showing promise for adaptive testing and efficient assessment in large-scale platforms.
• The integration of metadata and multi-modal features (e.g., question images and text) significantly improved performance in matrix completion and prediction tasks.
• The competition revealed that active learning and uncertainty-based sampling strategies were effective for optimizing question sequence selection, reducing the number of required responses for accurate predictions.