[Paper Review] Use and evaluation of simulation for software process education: a case study

Nauman bin Ali, Michael Unterkalmsteiner|arXiv (Cornell University)|Jul 24, 2023

Software Engineering Techniques and Practices4 citations

TL;DR

This study evaluates the use of software process simulation (SimSE) in a graduate project management course to improve students' ability to justify their choice of software development processes. Using an extended Evidence-Based Reasoning (EBR) framework, the study finds that SimSE significantly improved the strength of students' arguments in post-course evaluations, though assessment remains time-intensive compared to traditional methods.

ABSTRACT

Software Engineering is an applied discipline and concepts are difficult to grasp only at a theoretical level alone. In the context of a project management course, we introduced and evaluated the use of software process simulation (SPS) based games for improving students' understanding of software development processes. The effects of the intervention were measured by evaluating the students' arguments for choosing a particular development process. The arguments were assessed with the Evidence-Based Reasoning framework, which was extended to assess the strength of an argument. The results indicate that students generally have difficulty providing strong arguments for their choice of process models. Nevertheless, the assessment indicates that the intervention of the SPS game had a positive impact on the students' arguments. Even though the illustrated argument assessment approach can be used to provide formative feedback to students, its use is rather costly and cannot be considered a replacement for traditional assessments.

Motivation & Objective

• To address students' difficulty in justifying software process model choices in project management courses.
• To investigate whether software process simulation (SPS) enhances students' understanding of process implications.
• To evaluate the effectiveness of the Evidence-Based Reasoning (EBR) framework in assessing argument strength for process selection.
• To explore the feasibility and impact of using SimSE in a real-world course setting rather than a controlled experiment.
• To provide formative feedback on argument quality using a detailed analytical framework.

Proposed method

• Implemented SimSE, a software process simulation game, as a pedagogical intervention in a graduate-level Applied Software Project Management (ASPM) course.
• Collected student reports on process model selection at two points: beginning (project management plan) and end (post-mortem) of the course.
• Applied an extended Evidence-Based Reasoning (EBR) framework to assess argument strength, analyzing components: premise, rule, and evidence/data.
• Classified arguments as strong, weak, or unsound based on completeness and logical coherence of EBR components.
• Used chi-square tests of independence to compare argument quality between course years (2012 vs. 2013), with and without SimSE.
• Conducted qualitative analysis of argument components to identify gaps (e.g., missing data analysis) for formative feedback.

Experimental results

Research questions

• RQ1 Does the use of software process simulation (SimSE) improve students’ ability to construct strong, evidence-based arguments for selecting a software process model?
• RQ2 How does the argument quality for process model selection differ between course instances with and without SimSE intervention?
• RQ3 To what extent can the EBR framework be used to assess and provide feedback on students’ reasoning in software process selection?
• RQ4 What are the practical challenges of using the EBR framework for assessing student arguments in a real course setting?

Key findings

• The chi-square test showed a statistically significant difference in argument quality for the post-mortem assignment (χ² = 8.608, df = 2, p-value = 0.009), indicating improved argument strength in 2013 after SimSE use.
• Students in 2013 showed a higher proportion of strong arguments in the post-course individual assignment compared to 2012, suggesting a positive impact of SimSE on understanding.
• Despite the intervention, most assignments still contained weak or unsound arguments, particularly due to missing data analysis or insufficient evidence.
• The EBR framework successfully identified specific argumentation gaps (e.g., lack of data interpretation), enabling targeted formative feedback.
• All groups in 2013 selected lightweight, people-centric process models (e.g., Scrum), whereas in 2012 most groups chose plan-driven models, indicating improved contextual understanding.
• Argument assessment using EBR was time-intensive, requiring 10–30 minutes per report, highlighting a cost barrier compared to simpler rubric-based assessments.