[Paper Review] Teaching Python programming with automatic assessment and feedback provision

Hans Fangohr, Neil S. O’Brien+2 more|arXiv (Cornell University)|Sep 11, 2015

Teaching and Learning Programming23 references7 citations

TL;DR

This paper presents a scalable, automated feedback system for teaching Python programming using unit testing frameworks, where students submit code that is instantly evaluated against predefined test cases and receive immediate, detailed feedback via email. The system significantly reduces staff workload, supports iterative learning, and improves student engagement and performance in large cohorts, as demonstrated in a 425-student course with high satisfaction and efficient assessment throughput.

ABSTRACT

We describe a method of automatic feedback provision for students learning programming and computational methods in Python. We have implemented, used and refined this system since 2009 for growing student numbers, and summarise the design and experience of using it. The core idea is to use a unit testing framework: the teacher creates a set of unit tests, and the student code is tested by running these tests. With our implementation, students typically submit work for assessment, and receive feedback by email within a few minutes after submission. The choice of tests and the reporting back to the student is chosen to optimise the educational value for the students. The system very significantly reduces the staff time required to establish whether a student's solution is correct, and shifts the emphasis of computing laboratory student contact time from assessing correctness to providing guidance. The self-paced nature of the automatic feedback provision supports a student-centred learning approach. Students can re-submit their work repeatedly and iteratively improve their solution, and enjoy using the system. We include an evaluation of the system and data from using it in a class of 425 students.

Motivation & Objective

• To address the growing challenge of assessing programming assignments in large undergraduate and postgraduate classes efficiently and consistently.
• To reduce manual marking burden on teaching staff by automating correctness checks through unit testing frameworks.
• To support student-centered, self-paced learning by enabling rapid, iterative submission and feedback cycles.
• To shift teaching staff focus from basic correctness checks to higher-level guidance on code quality, readability, and design.
• To evaluate the system’s impact on student learning, satisfaction, and scalability in real-world university courses.

Proposed method

• The system uses a unit testing framework where instructors define test cases for each programming exercise, which are automatically executed against student submissions.
• Student code is submitted via a web interface, and the system runs the predefined unit tests to check correctness, performance, and robustness.
• Feedback is generated automatically and delivered to students via email within minutes of submission, highlighting test failures and providing diagnostic information.
• The system is designed with loosely coupled processes and file-based job queues, enabling scalability and integration with existing platforms like Moodle.
• The feedback is optimized to maximize educational value by focusing on correctness and early error detection, while higher-level code quality is assessed manually.
• The system supports repeated submissions, enabling students to iteratively refine their solutions until they pass all tests.

Experimental results

Research questions

• RQ1 How effective is an automated unit testing system in reducing staff workload for assessing programming assignments in large classes?
• RQ2 To what extent does immediate, automated feedback improve student learning outcomes and engagement in introductory programming courses?
• RQ3 How does the system support self-paced, iterative learning and student autonomy in programming education?
• RQ4 What impact does automated feedback have on student satisfaction and perceived learning in large-scale programming courses?
• RQ5 Can the system be scaled and adapted for use across different institutions and programming languages beyond Python?

Key findings

• The system reduced staff time for assessing correctness from hours to minutes per submission, enabling efficient handling of 425 students in a single course.
• Students submitted an average of 4.5 times per exercise, indicating strong engagement with the iterative refinement model enabled by instant feedback.
• Student attendance in scheduled lab sessions declined significantly after the first two weeks, suggesting that many students completed exercises independently and earlier.
• The system allowed the course to scale from 85 to 425 students without increasing staff time for marking, demonstrating high scalability.
• Student feedback on the system was overwhelmingly positive, with learners reporting increased enjoyment and deeper understanding of programming concepts.
• The system’s architecture proved robust and extensible, supporting integration with external platforms like Moodle and enabling reuse across institutions.