[Paper Review] Limits of an AI program for solving college math problems

Ernest Davis|arXiv (Cornell University)|Aug 14, 2022

Artificial Intelligence in Education41 citations

TL;DR

The paper critiques Drori et al. 2022, arguing that Sympy performs the problem solving while the neural component mostly formats input; it questions the 81% success claim and highlights methodological and interpretive issues.

ABSTRACT

Drori et al. (2022) report that "A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level ... [It] automatically answers 81\% of university-level mathematics problems." The system they describe is indeed impressive; however, the above description is very much overstated. The work of solving the problems is done, not by a neural network, but by the symbolic algebra package Sympy. Problems of various formats are excluded from consideration. The so-called "explanations" are just rewordings of lines of code. Answers are marked as correct that are not in the form specified in the problem. Most seriously, it seems that in many cases the system uses the correct answer given in the test corpus to guide its path to solving the problem.

Motivation & Objective

• Assess how the system solves university-level math problems and what components are responsible.
• Evaluate the validity of the claimed 81% success rate and the learning setup (zero-shot vs few-shot).
• Identify mismatches between problem formats, problem statements, and provided explanations or outputs.
• Examine the nature of explanations and whether they meaningfully illuminate problem solving.

Proposed method

• Descriptive critique of the problem-solving pipeline and the role of Sympy in the solution process.
• Comparison of claimed performance with results from GPT-3 and other baselines.
• Examination of problem formats, input adaptations, and alignment of answers to problem requirements.
• Analysis of the reported explanations and their usefulness for learning or understanding.

Experimental results

Research questions

• RQ1 What components actually perform the mathematical solving in the system (neural network vs symbolic math package)?
• RQ2 Does the reported 81% success rate accurately reflect the system's capabilities across problem types and formats?
• RQ3 Are the claimed explanations and few-shot additions methodologically sound and genuinely instructional?
• RQ4 How do problem formats and evaluation criteria affect the reported outcomes?
• RQ5 To what extent does the system rely on test corpus answers or scoring artifacts in guiding its solutions?

Key findings

• The actual solving work is done by Sympy, not the neural network.
• The system cannot solve problems involving images or proofs, challenging the 81% claim.
• Some outputs are not in the requested form of the original problems.
• Explanations consist largely of line-by-line code descriptions rather than meaningful problem explanations.
• There are concerns about how zero-shot and few-shot learning are triggered and whether the process relies on leveraging correct answers from the corpus.
• The paper's authorship and methodological details raise questions about sloppiness and potential misrepresentation.