[Paper Review] From Human Days to Machine Seconds: Automatically Answering and Generating Machine Learning Final Exams

Iddo Drori, Sarah Zhang+14 more|arXiv (Cornell University)|Jun 11, 2022

Online Learning and Analytics4 citations

TL;DR

This paper introduces a benchmark dataset of 646 machine learning final exam question parts from top-tier universities and demonstrates that large language models (LLMs) can solve these exams at a human level using few-shot prompting and chain-of-thought reasoning. The models generate new, high-quality exam questions in seconds, reducing instructor workload from days to seconds.

ABSTRACT

A final exam in machine learning at a top institution such as MIT, Harvard, or Cornell typically takes faculty days to write, and students hours to solve. We demonstrate that large language models pass machine learning finals at a human level, on finals available online after the models were trained, and automatically generate new human-quality final exam questions in seconds. Previous work has developed program synthesis and few-shot learning methods to solve university-level problem set questions in mathematics and STEM courses. In this work, we develop and compare methods that solve final exams, which differ from problem sets in several ways: the questions are longer, have multiple parts, are more complicated, and span a broader set of topics. We curate a dataset and benchmark of questions from machine learning final exams available online and code for answering these questions and generating new questions. We show how to generate new questions from other questions and course notes. For reproducibility and future research on this final exam benchmark, we use automatic checkers for multiple-choice, numeric, and questions with expression answers. We perform ablation studies comparing zero-shot learning with few-shot learning and chain-of-thought prompting using GPT-3, OPT, Codex, and ChatGPT across machine learning topics and find that few-shot learning methods perform best. We highlight the transformative potential of language models to streamline the writing and solution of large-scale assessments, significantly reducing the workload from human days to mere machine seconds. Our results suggest that rather than banning large language models such as ChatGPT in class, instructors should teach students to harness them by asking students meta-questions about correctness, completeness, and originality of the responses generated, encouraging critical thinking in academic studies.

Motivation & Objective

• To address the gap in evaluating LLMs on comprehensive, multi-part final exams, which differ significantly from standard problem sets in breadth, complexity, and depth.
• To curate a reproducible benchmark of real machine learning final exam questions from MIT, Harvard, and Cornell, covering diverse topics and question types.
• To develop and evaluate automated methods for answering and generating high-quality, human-like final exam questions using LLMs.
• To assess the quality and pedagogical suitability of machine-generated questions through student surveys comparing them to human-written questions.
• To demonstrate that LLMs can reduce instructor workload from human days to machine seconds in course assessment creation and grading.

Proposed method

• Curated a dataset of 646 question parts from final exams of three top-tier machine learning courses at MIT, Harvard, and Cornell, covering theory, math, coding, and conceptual reasoning.
• Implemented automatic checkers for multiple-choice, numeric, and expression-based answers to enable reproducible evaluation of model outputs.
• Evaluated multiple LLMs (GPT-3, OPT, Codex, ChatGPT) using zero-shot, few-shot, and chain-of-thought prompting across diverse ML topics and semesters.
• Used text-similarity embedding (text-similarity-babbage-001) to find the closest training examples for few-shot prompting, ensuring context relevance.
• Generated new questions by prompting LLMs with existing questions and course notes, using cosine similarity to ensure topical consistency.
• Fixed model hyperparameters (temperature=0, top_p=0, no stop sequences) for deterministic, reproducible results during evaluation; increased diversity (temperature=0.1) for question generation.

Experimental results

Research questions

• RQ1 Can large language models solve university-level machine learning final exams at a human-level performance, despite their length, complexity, and multi-part structure?
• RQ2 How do different prompting strategies—zero-shot, few-shot, and chain-of-thought—compare in solving complex, multi-concept final exam questions?
• RQ3 To what extent can LLMs generate new, high-quality, and pedagogically appropriate final exam questions that are indistinguishable from human-written ones?
• RQ4 How do students perceive the quality, difficulty, and authenticity of machine-generated final exam questions compared to human-written ones?
• RQ5 Can automatic checkers reliably validate LLM-generated answers for diverse question types (multiple-choice, numeric, expression-based) in a standardized way?

Key findings

• Few-shot learning with chain-of-thought prompting achieved the best performance across all machine learning topics and semesters, outperforming zero-shot and standard few-shot methods.
• Students correctly identified only 56.11% of human-written questions as human-written, and 45% of machine-generated questions as machine-generated, indicating strong human-like realism.
• Machine-generated questions were rated as slightly easier (mean difficulty 2.55) than human-written questions (mean difficulty 2.85), but both were considered appropriate for the course 82.6% and 85.0% of the time, respectively.
• The survey results show that machine-generated questions are perceived as equally appropriate and of comparable quality to human-written questions, with no significant difference in perceived authenticity.
• The entire process of answering or generating a question part takes only a few seconds, reducing instructor workload from human days to machine seconds.
• The benchmark and dataset are reproducible, with deterministic outputs ensured by fixing model hyperparameters (temperature=0, top_p=0) during evaluation.