[Paper Review] A conversation around the analysis of the SiP effort estimation dataset

Derek M. Jones, Stephen Cullum|arXiv (Cornell University)|Jan 6, 2019

Software Engineering Research8 references4 citations

TL;DR

This paper presents a conversational analysis of the SiP effort estimation dataset—10,100 task estimates from 22 developers across 20 projects over 10+ years—revealing that estimation accuracy is significantly influenced by individual developers, project context, and round-number bias. Despite improved estimation practice, regression models showed no noticeable improvement, highlighting the limits of statistical modeling in capturing human estimation behavior in agile environments.

ABSTRACT

The analysis of over ten years of commercial development using Agile (10,100 unique task estimates made by 22 developers, under 20 project codes) is documented via a conversation involving the data analyst and a director of the company that created the SiP dataset. Factors found to influence task implementation effort estimation accuracy include the person making the estimate, the project involved, and the propensity to use round numbers. Any improvement in estimation accuracy, with practice, did not noticeably improve regression models fitted.

Motivation & Objective

• To investigate the factors influencing software task effort estimation accuracy using a large, real-world dataset from a commercial agile development environment.
• To explore how individual developer behavior, project-specific dynamics, and estimation practices affect estimation accuracy over time.
• To assess whether improved estimation practice leads to measurable improvements in regression models of effort estimation.
• To understand the role of client involvement and organizational practices in shaping estimation behavior and outcomes.
• To provide actionable insights for software organizations on data collection and estimation improvement through empirical analysis.

Proposed method

• Conducted iterative data analysis through a dialogue between a domain expert (Stephen Cullum) and a data analyst (Derek Jones), integrating domain knowledge with statistical techniques.
• Analyzed 12,299 rows of anonymized data from the SiP dataset, including task estimates, actual effort, developer IDs, project codes, and timestamps from 2004 to 2014.
• Used R for statistical modeling and visualization, with code snippets integrated into the analysis narrative to support findings.
• Evaluated estimation accuracy using metrics such as relative error and bias, comparing individual developers, projects, and time periods.
• Explored patterns in round-number usage and their correlation with estimation inaccuracy.
• Assessed the impact of client involvement and release planning practices on estimation behavior and outcomes.

Experimental results

Research questions

• RQ1 What factors significantly influence the accuracy of software task effort estimation in an agile environment?
• RQ2 How does individual developer behavior affect estimation accuracy, and does experience lead to measurable improvement?
• RQ3 To what extent does the use of round numbers impact estimation inaccuracy?
• RQ4 Does improved estimation practice over time result in better-performing regression models?
• RQ5 How does client involvement and project phase (development vs. support) affect estimation behavior and outcomes?

Key findings

• Estimation accuracy was significantly influenced by the individual developer, with notable variation in bias and precision across the 22 developers.
• Project-specific factors, including client involvement and project phase, had a strong impact on estimation behavior and outcomes.
• A strong propensity to use round numbers was observed, contributing to systematic inaccuracy in estimates.
• Despite observed improvements in estimation practice over time, regression models did not show noticeable performance gains.
• In the support phase (post-2008), overestimates were more common and accepted by clients, suggesting a strategic shift in estimation behavior.
• Client involvement during the development phase drove higher estimation accuracy, while reduced involvement in support phases correlated with overestimation.