[Paper Review] An Empirical Comparative Study of Checklist based and Ad Hoc Code Reading Techniques in a Distributed Groupware Environment

Olalekan S. Akinola, Adenike Osofisan|ArXiv.org|Sep 23, 2009

Software Engineering Research27 references3 citations

TL;DR

This empirical study compares checklist-based and ad hoc code reading techniques in a distributed groupware environment using 20 computer science undergraduates. Results show no significant differences in defect detection effectiveness, time spent, or false positive rates between the two methods, suggesting both are equally effective in distributed settings.

ABSTRACT

Software inspection is a necessary and important tool for software quality assurance. Since it was introduced by Fagan at IBM in 1976, arguments exist as to which method should be adopted to carry out the exercise, whether it should be paper based or tool based, and what reading technique should be used on the inspection document. Extensive works have been done to determine the effectiveness of reviewers in paper based environment when using ad hoc and checklist reading techniques. In this work, we take the software inspection research further by examining whether there is going to be any significant difference in defect detection effectiveness of reviewers when they use either ad hoc or checklist reading techniques in a distributed groupware environment. Twenty final year undergraduate students of computer science, divided into ad hoc and checklist reviewers groups of ten members each were employed to inspect a medium sized java code synchronously on groupware deployed on the Internet. The data obtained were subjected to tests of hypotheses using independent T test and correlation coefficients. Results from the study indicate that there are no significant differences in the defect detection effectiveness, effort in terms of time taken in minutes and false positives reported by the reviewers using either ad hoc or checklist based reading techniques in the distributed groupware environment studied.

Motivation & Objective

• To investigate whether checklist-based code reading is more effective than ad hoc reading in a distributed groupware environment.
• To evaluate the impact of reading technique on defect detection effectiveness in a collaborative, tool-supported setting.
• To compare the time effort and false positive rates between checklist and ad hoc review methods in a distributed context.
• To extend prior research on code inspection from paper-based to distributed, groupware-based environments.
• To provide evidence-based guidance on selecting code reading techniques for distributed software development teams.

Proposed method

• Twenty final-year computer science students were divided into two groups: one using checklist-based reading and the other using ad hoc reading.
• A medium-sized Java codebase was inspected synchronously via Internet-deployed groupware tools.
• Defect detection effectiveness, time spent (in minutes), and false positive reports were recorded for each reviewer.
• Independent t-tests were used to compare the means of defect detection, effort, and false positives between the two groups.
• Product-moment correlation coefficients were calculated to assess relationships between variables.
• Data were collected and analyzed to test hypotheses regarding the superiority of one technique over the other.

Experimental results

Research questions

• RQ1 Is there a significant difference in defect detection effectiveness between checklist-based and ad hoc code reading in a distributed groupware environment?
• RQ2 Does the use of checklists reduce the time required for code inspection compared to ad hoc reading in a distributed setting?
• RQ3 Are false positive defect reports significantly different between checklist and ad hoc code reading techniques in this environment?
• RQ4 Is there a statistically significant difference in reviewer performance between the two techniques when using groupware tools?
• RQ5 Does the distributed nature of the groupware environment alter the relative effectiveness of checklist versus ad hoc code reading?

Key findings

• There was no statistically significant difference in defect detection effectiveness between checklist-based and ad hoc code reading techniques (p > 0.05).
• No significant difference was found in the time taken to complete inspections between the two techniques (p > 0.05).
• The number of false positive defect reports did not differ significantly between checklist and ad hoc reviewers (p > 0.05).
• Correlation analysis showed no strong linear relationship between defect detection and time spent or false positives.
• The results suggest that both techniques perform equivalently in terms of key inspection metrics in the studied distributed groupware environment.
• The study provides no evidence that checklist-based reading offers measurable advantages over ad hoc reading in this specific context.