[Paper Review] Computing with Codio at Coventry University: Online virtual Linux boxes and automated formative feedback

David Croft, Matthew England|arXiv (Cornell University)|Dec 10, 2018

Online Learning and Analytics3 references4 citations

TL;DR

This paper presents Coventry University's integration of Codio—a cloud-based platform offering online virtual Linux environments and automated formative feedback—into its undergraduate programming curriculum. The system provides instant, consistent feedback through automated testing of student code, resulting in improved student performance, higher satisfaction, and enhanced engagement tracking, despite challenges related to institutional regulations and initial development overhead.

ABSTRACT

We describe our experience using Codio at Coventry University in our undergraduate programming curriculum. Codio provides students with online virtual Linux boxes, and allows staff to equip these with guides written in markdown and supplemental tasks that provide automated feedback. The use of Codio has coincided with a steady increase in student performance and satisfaction as well as far greater data on student engagement and performance.

Motivation & Objective

• To address inconsistent, delayed, and resource-intensive formative feedback in large-scale programming modules.
• To improve student engagement and performance in introductory programming courses with 480+ first-year students.
• To provide consistent, instant feedback across diverse teaching staff and student skill levels.
• To enable better monitoring of student progress through automated tracking of time-on-task and activity completion.

Proposed method

• Deployment of Codio's cloud-based virtual Linux machines as a standardized development environment for all students.
• Creation of interactive, markdown-formatted guides with embedded coding tasks that trigger automated feedback.
• Implementation of automated testing via unit tests to evaluate student code submissions in real time.
• Integration of Codio with the university's existing Virtual Learning Environment (VLE) for course management, despite institutional restrictions.
• Use of Codio's analytics to track student engagement, including time spent per week and task completion rates.
• Adoption of Coderunner as a complementary tool for automated summative assessment using unit tests within the university's own Moodle infrastructure.

Experimental results

Research questions

• RQ1 Can automated formative feedback in a cloud-based virtual environment improve student performance in large undergraduate programming modules?
• RQ2 How does the use of Codio affect student engagement and satisfaction compared to traditional lab-based teaching?
• RQ3 To what extent does automated feedback reduce the burden on academic staff while maintaining consistency in assessment?
• RQ4 What institutional and technical challenges arise when integrating third-party platforms like Codio into a university's existing educational infrastructure?

Key findings

• Average student performance improved significantly, with quiz scores rising from 57% to 63% and exam scores from 51% to 59% after Codio implementation.
• Student satisfaction increased from 82% to 89%, with 84% of free-text comments about Codio being positive.
• Engagement tracking revealed detailed time-on-task data, with individual student activity patterns visualized over time.
• Students with prior programming experience were often critical of Codio due to restrictions on using preferred IDEs, highlighting a trade-off between standardization and flexibility.
• Despite technical and administrative hurdles, Codio enabled consistent, scalable feedback across 12 different staff members and 40 lab sessions.
• The university is now extending automated assessment to summative work using Coderunner, a locally hosted open-source alternative to Codio’s assessment engine.