[Paper Review] Why Software Effort Estimation Needs SBSE

Tianpei Xia, Jianfeng Chen+3 more|arXiv (Cornell University)|Apr 2, 2018

Software Engineering Research48 references3 citations

TL;DR

This paper proposes OIL, a CPU-light SBSE framework using differential evolution to automatically configure effort estimation models for software projects. Tested on 945 projects, OIL outperformed widely used methods like ABE0 and ATLM with just 40–160 evaluations, achieving significantly lower estimation errors in under six minutes median runtime on a standard laptop.

ABSTRACT

Industrial practitioners now face a bewildering array of possible configurations for effort estimation. How to select the best one for a particular dataset? This paper introduces OIL (short for optimized learning), a novel configuration tool for effort estimation based on differential evolution. When tested on 945 software projects, OIL significantly improved effort estimations, after exploring just a few configurations (just a few dozen). Further OIL's results are far better than two methods in widespread use: estimation-via-analogy and a recent state-of-the-art baseline published at TOSEM'15 by Whigham et al. Given that the computational cost of this approach is so low, and the observed improvements are so large, we conclude that SBSE should be a standard component of software effort estimation.

Motivation & Objective

• Address the challenge of selecting the best effort estimation method for a given dataset amid thousands of possible configurations.
• Reduce the computational burden of SBSE in effort estimation by developing a CPU-light approach that avoids the high evaluation costs of traditional evolutionary algorithms.
• Demonstrate that automated configuration via SBSE can significantly improve estimation accuracy compared to widely used baseline methods like ATLM and ABE0.
• Show that SBSE is both practical and necessary for effort estimation, even with limited computational resources and small datasets.
• Investigate whether floor effects or inherent data limitations constrain the performance of effort estimation models, and whether SBSE can still yield substantial gains under such conditions.

Proposed method

• OIL employs differential evolution (DE) as the search algorithm to explore the configuration space of analogy-based effort estimation (ABE) methods.
• The method evaluates candidate configurations using 10-fold cross-validation to estimate model performance on unseen data.
• OIL systematically explores thousands of combinations of ABE parameters (e.g., similarity metrics, normalization methods, outlier handling) to identify the optimal configuration.
• The search is guided by DE’s mutation, crossover, and selection mechanisms, with default parameter settings from Storn et al. to ensure stability and efficiency.
• The framework is designed for low computational cost: median runtime across 945 projects was just six minutes on a standard 8GB, 3GHz desktop.
• OIL is implemented as a reusable Python pip package to promote adoption and extensibility in the research and practitioner communities.

Experimental results

Research questions

• RQ1 RQ1: Can effort estimation ignore SBSE? Is tuning avoidable if only a few methods are consistently best across datasets?
• RQ2 RQ2: Is SBSE too computationally expensive to apply in practice, especially for developers and business users?
• RQ3 RQ3: Does SBSE produce better effort estimates than widely used methods like ABE0 and ATLM?
• RQ4 RQ4: Are there floor effects in effort estimation that limit performance improvements, regardless of method complexity?
• RQ5 RQ5: Does increasing the number of evaluations (e.g., from 40 to 160) lead to better estimation performance?

Key findings

• OIL significantly outperformed both ABE0 and ATLM across 945 software projects, with estimation errors that were statistically and substantially lower.
• The median runtime for OIL’s 10-fold cross-validation across all projects was just six minutes on a standard 8GB, 3GHz desktop machine.
• Only 40 evaluations of candidate configurations were sufficient to achieve optimal or near-optimal performance, indicating that SBSE is computationally feasible even on low-end hardware.
• Increasing the number of evaluations from 40 to 160 did not yield any further improvement, suggesting that the search space is efficiently explored with minimal effort.
• The best-performing estimation method varied significantly across datasets, demonstrating that no single method is universally optimal—hence the need for automated configuration via SBSE.
• Evidence of floor effects was observed: some datasets showed limited signal, and random searchers sometimes performed nearly as well as DE, suggesting inherent performance ceilings in certain data distributions.