[Paper Review] Play to Grade: Testing Coding Games as Classifying Markov Decision Process

Allen Nie, Emma Brunskill+1 more|arXiv (Cornell University)|Oct 27, 2021

Reinforcement Learning in Robotics38 references4 citations

TL;DR

This paper proposes Play to Grade, a method that uses reinforcement learning to automatically assess interactive coding assignments by treating them as Markov Decision Processes (MDPs). By training an agent to explore and sample differential trajectories from student programs, a classifier can then distinguish between correct and buggy programs with 93.4–94.0% accuracy, significantly outperforming code-text-only baselines.

ABSTRACT

Contemporary coding education often presents students with the task of developing programs that have user interaction and complex dynamic systems, such as mouse based games. While pedagogically compelling, there are no contemporary autonomous methods for providing feedback. Notably, interactive programs are impossible to grade by traditional unit tests. In this paper we formalize the challenge of providing feedback to interactive programs as a task of classifying Markov Decision Processes (MDPs). Each student's program fully specifies an MDP where the agent needs to operate and decide, under reasonable generalization, if the dynamics and reward model of the input MDP should be categorized as correct or broken. We demonstrate that by designing a cooperative objective between an agent and an autoregressive model, we can use the agent to sample differential trajectories from the input MDP that allows a classifier to determine membership: Play to Grade. Our method enables an automatic feedback system for interactive code assignments. We release a dataset of 711,274 anonymized student submissions to a single assignment with hand-coded bug labels to support future research.

Motivation & Objective

• To address the lack of automated feedback for interactive programming assignments in online education platforms.
• To overcome the limitations of traditional unit testing and code-text analysis in grading dynamic, user-interactive programs.
• To develop a scalable, generalizable method for identifying bugs in student-written interactive programs without relying on static code analysis.
• To enable feedback systems that can detect both functional correctness and subtle behavioral bugs in real-time.
• To support future research by releasing a large-scale dataset of 711,274 anonymized student submissions with hand-coded bug labels.

Proposed method

• Formalize interactive coding assignments as MDPs where each student program defines a unique MDP with state, action, and reward dynamics.
• Train a cooperative agent to play the student’s program and sample trajectories that highlight differences between correct and broken behavior.
• Use a contrastive learning framework (Contrastive HoareLSTM) to learn embeddings of trajectories that distinguish between correct and buggy MDPs.
• Apply a multi-distance voting mechanism using 10 contrastive distance functions to improve robustness and generalization of the final classification.
• Leverage a pre-trained 'play-to-win' agent to generate high-impact, differential trajectories that expose latent bugs.
• Design an OpenAI Gym-compatible environment and release the Bounce dataset to support reproducibility and future research.

Experimental results

Research questions

• RQ1 Can reinforcement learning be used to automatically detect bugs in interactive student programs without relying on code text analysis?
• RQ2 Can a cooperative agent–classifier framework effectively identify subtle behavioral differences between correct and broken MDPs in student games?
• RQ3 How does the performance of trajectory-based classification compare to traditional code-text-based grading methods on real-world coding assignments?
• RQ4 To what extent can the Play to Grade framework generalize to non-game interactive programming assignments with user input?
• RQ5 Can iterative training improve the robustness and accuracy of the classifier in identifying diverse bug patterns?

Key findings

• The Play to Grade method achieves 93.4–94.0% accuracy in classifying student programs as correct or broken, representing a 19–25 percentage point improvement over code-text-only baselines.
• The contrastive learning approach with trajectory sampling significantly outperforms majority-class baselines, which achieve only 50.0% accuracy.
• The method achieves 99.6% recall on the body distribution and 97.6% on the tail distribution, indicating strong detection of truly broken programs.
• The classifier maintains high precision (88.6–91.0%) while achieving near-perfect recall, minimizing false positives.
• The framework generalizes well across different program distributions, including both common and rare student solutions.
• The release of the Bounce dataset with 711,274 submissions and hand-coded bug labels provides a valuable benchmark for future research in automated feedback systems.