[Paper Review] Exploring the MIT Mathematics and EECS Curriculum Using Large Language Models

Sarah Zhang, Samuel Florin+13 more|arXiv (Cornell University)|Jun 15, 2023

Oil and Gas Production Techniques18 citations

TL;DR

The paper builds a 4,550-question dataset from MIT Math and EECS, evaluating LLMs’ ability to meet graduation requirements and revealing curriculum insights through automatic grading and question embeddings.

ABSTRACT

We curate a comprehensive dataset of 4,550 questions and solutions from problem sets, midterm exams, and final exams across all MIT Mathematics and Electrical Engineering and Computer Science (EECS) courses required for obtaining a degree. We evaluate the ability of large language models to fulfill the graduation requirements for any MIT major in Mathematics and EECS. Our results demonstrate that GPT-3.5 successfully solves a third of the entire MIT curriculum, while GPT-4, with prompt engineering, achieves a perfect solve rate on a test set excluding questions based on images. We fine-tune an open-source large language model on this dataset. We employ GPT-4 to automatically grade model responses, providing a detailed performance breakdown by course, question, and answer type. By embedding questions in a low-dimensional space, we explore the relationships between questions, topics, and classes and discover which questions and classes are required for solving other questions and classes through few-shot learning. Our analysis offers valuable insights into course prerequisites and curriculum design, highlighting language models' potential for learning and improving Mathematics and EECS education.

Motivation & Objective

• Motivate evaluating LLMs against MIT Mathematics and EECS graduation requirements.
• Assemble a comprehensive dataset of problems and solutions from MIT courses.
• Assess GPT-3.5 and GPT-4 performance on solving and grading MIT questions.
• Explore curriculum relationships via low-dimensional embeddings and few-shot learning.
• Provide insights for curriculum design and potential educational applications of LLMs.

Proposed method

• Assemble a dataset of 4,550 questions and solutions from MIT Math and EECS courses.
• Evaluate GPT-3.5 and GPT-4 on solving problems across the curriculum.
• Fine-tune an open-source LLM on the dataset.
• Use GPT-4 to automatically grade model responses and breakdown performance by course, question, and answer type.
• Embed questions in a low-dimensional space to analyze relationships between questions, topics, and classes.
• Apply few-shot learning to assess prerequisites and cross-class solvability.

Experimental results

Research questions

• RQ1 Can LLMs solve problems across MIT Mathematics and EECS to meet graduation requirements?
• RQ2 How well do GPT-3.5 and GPT-4 perform on solving the MIT problem-set data?
• RQ3 Can fine-tuning an open-source LLM improve alignment with MIT problem-solving tasks?
• RQ4 What do embedding-based analyses reveal about prerequisite structures and curriculum design?
• RQ5 Can automatic grading via GPT-4 provide detailed, course-level performance insights?

Key findings

• GPT-3.5 solves about one-third of the MIT curriculum.
• GPT-4 achieves a perfect solve rate on a test set excluding image-based questions with prompt engineering.
• Fine-tuning an open-source LLM on the dataset is feasible and informative.
• GPT-4 can automatically grade model responses, delivering performance breakdowns by course, question, and answer type.
• Low-dimensional embeddings reveal relationships among questions, topics, and classes and help identify prerequisite structures through few-shot learning.
• The analysis offers insights into curriculum design and the role of LLMs in learning and improving Mathematics and EECS education.