[Paper Review] On How Developers Test Open Source Software Systems

Andy Zaidman, Bart Van Rompaey+2 more|arXiv (Cornell University)|May 24, 2007

Software Engineering Research24 references3 citations

TL;DR

This study investigates test practices in open source software by analyzing version control, code coverage, and size metrics across three projects. It finds that testing activities typically begin late in development and are phased, with no consistent increase before releases, despite varying test coverage and codebase sizes.

ABSTRACT

Engineering software systems is a multidisciplinary activity, whereby a number of artifacts must be created - and maintained - synchronously. In this paper we investigate whether production code and the accompanying tests co-evolve by exploring a project's versioning system, code coverage reports and size-metrics. Three open source case studies teach us that testing activities usually start later on during the lifetime and are more "phased", although we did not observe increasing testing activity before releases. Furthermore, we note large differences in the levels of test coverage given the proportion of test code.

Motivation & Objective

• Understand how testing practices co-evolve with production code in open source systems.
• Investigate whether testing activity increases before software releases.
• Analyze the relationship between test code volume and test coverage across project lifecycles.
• Explore the temporal patterns of testing activities using versioning and coverage data.
• Assess the extent to which testing is integrated early versus late in the development process.

Proposed method

• Analyzed version control histories of three open source projects to track code and test changes over time.
• Collected and evaluated code coverage reports to measure test effectiveness and coverage trends.
• Extracted size metrics (e.g., lines of code, test lines) to compare test effort relative to production code.
• Mapped the evolution of test and production code across development milestones and release cycles.
• Identified phases of testing activity by analyzing temporal clustering of test commits.
• Used statistical analysis to correlate test activity with release timelines and code changes.

Experimental results

Research questions

• RQ1 Do testing activities increase before software releases in open source projects?
• RQ2 How does the proportion of test code relate to actual test coverage across the project lifecycle?
• RQ3 When do testing activities typically begin relative to the start of development?
• RQ4 Is there a consistent pattern of test development that aligns with major development phases?
• RQ5 How do test coverage and test code volume evolve in relation to production code changes?

Key findings

• Testing activities generally begin later in the project lifecycle rather than being integrated from the start.
• Test development is often phased, with clusters of test commits occurring in distinct intervals rather than a steady pace.
• No consistent increase in testing activity was observed before releases, contradicting assumptions of pre-release testing surges.
• Significant variation exists in test coverage across projects, even with similar proportions of test code.
• The ratio of test code to production code does not reliably predict test coverage levels.
• Co-evolution between production code and tests is weak, indicating that tests are often added reactively rather than proactively.