[Paper Review] Computing with CodeRunner at Coventry University: Automated summative assessment of Python and C++ code

David Croft, Matthew England|arXiv (Cornell University)|Nov 25, 2019

Teaching and Learning Programming5 references4 citations

TL;DR

This paper evaluates CodeRunner, an open-source Moodle plugin, for automated summative assessment of Python and C++ code in a first-year computing curriculum at Coventry University. By replacing multiple-choice quizzes with executable code tests, the study shows improved assessment authenticity, higher student satisfaction with feedback (up 7%), and more accurate reflection of programming ability, though C++ assessment required custom SFINAE-based introspection to award partial credit for syntactically correct but logically flawed code.

ABSTRACT

CodeRunner is a free open-source Moodle plugin for automatically marking student code. We describe our experience using CodeRunner for summative assessment in our first year undergraduate programming curriculum at Coventry University. We use it to assess both Python3 and C++14 code (CodeRunner supports other languages also). We give examples of our questions and report on how key metrics have changed following its use at Coventry.

Motivation & Objective

• To address the limitations of multiple-choice assessments in measuring genuine programming ability in first-year computing students.
• To implement and evaluate CodeRunner for automated, executable assessment of both Python and C++ code in a summative context.
• To improve feedback quality and student satisfaction by replacing low-fidelity assessments with real code execution.
• To solve the challenge of awarding partial credit in C++ assessments where compilation failures prevent test execution.
• To develop and deploy a SFINAE-based introspection library to detect correct code structure despite minor syntax errors.

Proposed method

• Adopted CodeRunner as a Moodle plugin to automatically assess student-submitted Python and C++ code using unit tests.
• Designed assessment questions with multiple test cases, including hidden ones, to evaluate correctness and robustness.
• Implemented a custom C++ introspection library using SFINAE and preprocessor macros to detect function signatures and class structures.
• Used the introspection library to allow partial credit by identifying correct code structure even when compilation failed due to minor errors.
• Replaced traditional multiple-choice mid-term tests with executable code assessments in two first-year modules.
• Collected and analyzed student performance data, feedback satisfaction scores, and assessment outcomes across two academic years.

Experimental results

Research questions

• RQ1 Does replacing multiple-choice assessments with executable code testing via CodeRunner improve the accuracy of measuring student programming ability?
• RQ2 How does automated assessment with CodeRunner affect student satisfaction with feedback and assessment processes?
• RQ3 What challenges arise when using CodeRunner for C++ assessment, particularly regarding compilation failures preventing test execution?
• RQ4 Can SFINAE-based introspection techniques effectively identify correct code structure and award partial credit in C++ despite compilation errors?
• RQ5 What is the impact of automated assessment on pass rates and average marks compared to traditional assessment methods?

Key findings

• The average mark in CodeRunner assessments was lower than in previous multiple-choice tests, indicating that CodeRunner more accurately reflects actual programming ability.
• Student satisfaction with assessment and feedback increased by 7 percentage points to 90% following the introduction of CodeRunner.
• The final written exam for 4000CEM showed a 7% higher average mark and 10% higher pass rate among students who experienced CodeRunner assessment, suggesting improved learning outcomes.
• In 4003CEM, the initial drop in marks and satisfaction prompted the development of a SFINAE-based introspection library to better handle C++ compilation errors and award partial credit.
• The SFINAE-based introspection library successfully enabled detection of correct code structure and provided meaningful error messages, reducing the risk of zero marks for minor syntax errors.
• The use of CodeRunner led to a more authentic assessment environment, closely mirroring real-world programming with syntax checking and iterative testing.