[Paper Review] sk_p: a neural program corrector for MOOCs

Yewen Pu, Karthik Narasimhan+2 more|arXiv (Cornell University)|Jul 11, 2016

Software Testing and Debugging Techniques4 citations

TL;DR

This paper introduces sk_p, a neural program corrector for MOOCs that uses a seq2seq model trained on correct Python programs to generate and validate candidate fixes for syntactic and semantic errors. It achieves 29% correction rate on MOOC assignments, outperforming state-of-the-art methods that require manual error modeling by leveraging data-driven, generative correction without symbolic analysis or predefined search spaces.

ABSTRACT

We present a novel technique for automatic program correction in MOOCs, capable of fixing both syntactic and semantic errors without manual, problem specific correction strategies. Given an incorrect student program, it generates candidate programs from a distribution of likely corrections, and checks each candidate for correctness against a test suite. The key observation is that in MOOCs many programs share similar code fragments, and the seq2seq neural network model, used in the natural-language processing task of machine translation, can be modified and trained to recover these fragments. Experiment shows our scheme can correct 29% of all incorrect submissions and out-performs state of the art approach which requires manual, problem specific correction strategies.

Motivation & Objective

• To address the challenge of providing accurate, automated feedback for incorrect student programs in MOOCs, especially in introductory programming courses.
• To eliminate the need for manual, problem-specific error models required by prior symbolic program synthesis approaches.
• To develop a scalable, data-driven method that leverages large corpora of correct student submissions to train a neural model for program correction.
• To enable fast, real-time correction using a generative model that samples from likely corrections rather than enumerating a large search space.
• To demonstrate that a simple, local, syntactic model can outperform more complex semantic-aware models in the context of MOOC program correction.

Proposed method

• Train a modified seq2seq neural network on tokenized statements from correct student programs to learn common code fragments.
• Use a skipgram-style context window—only one statement before and after the error site—to generate candidate corrections locally.
• Generate candidate programs by sampling from the model’s learned distribution of likely corrections for each incorrect statement.
• Validate each candidate against a hand-crafted test suite to check correctness, using an enumerate-and-check strategy.
• Leverage the generative nature of the model to avoid the need for defining and ranking a large, explicit search space of patches.
• Train the model end-to-end on a corpus of correct programs, with correctness verified via the same test suite used for validation.

Experimental results

Research questions

• RQ1 Can a purely syntactic, local neural model effectively correct both syntactic and semantic errors in student programs without symbolic analysis?
• RQ2 To what extent can a data-driven, generative approach outperform symbolic methods that require manual error model specification?
• RQ3 Does a minimal context (one statement before and after) provide sufficient information for accurate program correction in MOOCs?
• RQ4 Can a model trained on correct programs generalize to fix incorrect programs that are not direct matches to any training example?
• RQ5 How does the performance of a neural correction system compare to state-of-the-art symbolic and clustering-based approaches in real-world MOOC settings?

Key findings

• sk_p corrects 29% of all incorrect student submissions across 7 Python programming assignments from MITx’s 6.00x course.
• On 3 shared benchmarks, sk_p achieves 35% average accuracy, slightly outperforming the state-of-the-art Autograder system, which achieves 30% accuracy with manual error modeling.
• The system operates in an average of 5.6 seconds per submission, demonstrating real-time feasibility for MOOC feedback.
• The model successfully identifies and corrects non-trivial semantic errors, such as incorrect loop structures, by generating semantically valid alternatives.
• The method discovers 'fresh' corrections—valid fixes not present in the training set—demonstrating generalization beyond simple template matching.
• Despite its simplicity and lack of semantic analysis, the local, token-level model performs competitively, showing that syntactic patterns in MOOC code are highly predictive for correction.