[Paper Review] Towards a Statistical Methodology to Evaluate Program Speedups and their Optimisation Techniques

Sid Touati|ArXiv.org|Feb 6, 2009

Software Reliability and Analysis Research1 references3 citations

TL;DR

This paper proposes a statistical methodology to evaluate program speedups and optimization techniques using confidence intervals and hypothesis testing to address the reproducibility crisis in performance evaluation. It emphasizes reporting confidence levels and using representative benchmark sets to ensure results are statistically rigorous and reproducible under similar conditions.

ABSTRACT

The community of program optimisation and analysis, code performance evaluation, parallelisation and optimising compilation has published since many decades hundreds of research and engineering articles in major conferences and journals. These articles study efficient algorithms, strategies and techniques to accelerate programs execution times, or optimise other performance metrics (MIPS, code size, energy/power, MFLOPS, etc.). Many speedups are published, but nobody is able to reproduce them exactly. The non-reproducibility of our research results is a dark point of the art, and we cannot be qualified as {\it computer scientists} if we do not provide rigorous experimental methodology. This article provides a first effort towards a correct statistical protocol for analysing and measuring speedups. As we will see, some common mistakes are done by the community inside published articles, explaining part of the non-reproducibility of the results. Our current article is not sufficient by its own to deliver a complete experimental methodology, further efforts must be done by the community to decide about a common protocol for our future experiences. Anyway, our community should take care about the aspect of reproducibility of the results in the future.

Motivation & Objective

• To address the widespread non-reproducibility of program performance results in computer science research.
• To establish a rigorous statistical protocol for evaluating speedups and optimization techniques in program execution.
• To highlight common statistical flaws in published performance evaluations that undermine reproducibility.
• To promote transparency by requiring disclosure of confidence levels and experimental details in performance studies.
• To encourage the community to adopt standardized, statistically sound experimental methodologies for future work.

Proposed method

• Uses confidence intervals for proportions to quantify the reliability of speedup measurements across benchmarks.
• Applies the Student's t-test and z-test for comparing mean execution times between original and optimized programs.
• Employs the formula $ n \geq z^2 \times \frac{C(1-C)}{r^2} $ to determine the minimum number of benchmarks needed for desired precision.
• Recommends using a 90% or 95% confidence level for statistical assertions, depending on application context (e.g., hard real-time vs. desktop).
• Stresses the importance of reporting all experimental details, including environment, input data, and configuration, to enable reproducibility.
• Advocates for formal algorithmic descriptions and open software to allow exact replication of results.

Experimental results

Research questions

• RQ1 Why are published program speedups often non-reproducible despite rigorous experimental claims?
• RQ2 How can statistical methods be used to quantify the reliability and confidence in reported program speedups?
• RQ3 What is the minimum number of benchmarks required to achieve a desired precision in speedup estimation?
• RQ4 How do variations in execution time due to hardware and system factors affect the validity of performance measurements?
• RQ5 What confidence level should be used in performance evaluation, and how does it vary across different application domains?

Key findings

• A 90% confidence interval for a speedup proportion of 17/30 (56.67%) spans from 40.27% to 71.87%, indicating high uncertainty with only 30 benchmarks.
• To achieve a 95% confidence level with a 5% margin of error, at least 378 benchmarks are required to estimate the true proportion of accelerated programs.
• The use of low sample sizes (e.g., n.C < 10) invalidates standard confidence interval calculations, requiring more complex statistical methods.
• Even with identical hardware and software, execution times vary significantly due to factors like cache effects, interrupts, and dynamic voltage scaling.
• Many published results are based on rare events or biased measurements, making them statistically surprising and hard to reproduce.
• Confidence levels should be explicitly declared in publications, with higher levels (e.g., 95%) required for safety-critical applications like hard real-time systems.