[Paper Review] A Comprehensive Survey of Benchmarks for Automated Improvement of Software's Non-Functional Properties

Aymeric Blot, Justyna Petke|arXiv (Cornell University)|Dec 16, 2022

Cloud Computing and Resource Management4 citations

TL;DR

This paper presents a comprehensive survey of benchmarks used in automated improvement of software's non-functional properties, analyzing 386 relevant papers from major repositories. It reveals that execution time is the most targeted property (62%), static analysis dominates (53%), and SPEC is the most popular benchmark (33 papers), while multi-objective optimization and diverse language support (e.g., Python, JavaScript) remain underexplored.

ABSTRACT

Performance is a key quality of modern software. Although recent years have seen a spike in research on automated improvement of software's execution time, energy, memory consumption, etc., there is a noticeable lack of standard benchmarks for such work. It is also unclear how such benchmarks are representative of current software. Furthermore, frequently non-functional properties of software are targeted for improvement one-at-a-time, neglecting potential negative impact on other properties. In order to facilitate more research on automated improvement of non-functional properties of software, we conducted a survey gathering benchmarks used in previous work. We considered 5 major online repositories of software engineering work: ACM Digital Library, IEEE Xplore, Scopus, Google Scholar, and ArXiV. We gathered 5000 publications (3749 unique), which were systematically reviewed to identify work that empirically improves non-functional properties of software. We identified 386 relevant papers. We find that execution time is the most frequently targeted property for improvement (in 62% of relevant papers), while multi-objective improvement is rarely considered (5%). Static approaches are prevalent (in 53% of papers), with exploratory approaches (evolutionary in 18% and non-evolutionary in 14% of papers) increasingly popular in the last 10 years. Only 40% of 386 papers describe work that uses benchmark suites, rather than single software, of those SPEC is most popular (covered in 33 papers). We also provide recommendations for choice of benchmarks in future work, noting, e.g., lack of work that covers Python or JavaScript. We provide all programs found in the 386 papers on our dedicated webpage at https://bloa.github.io/nfunc_survey/ We hope that this effort will facilitate more research on the topic of automated improvement of software's non-functional properties.

Motivation & Objective

• To identify and analyze benchmarks used in automated improvement of software's non-functional properties (NFPs), such as execution time, memory, and energy.
• To assess the prevalence and diversity of software targets, programming languages, and benchmark suites in existing empirical studies.
• To evaluate the types of approaches used—static, evolutionary, or exploratory—for NFP improvement and their adoption trends.
• To highlight gaps in current research, including underrepresentation of modern languages (e.g., Python, JavaScript), mobile platforms, and multi-objective optimization.
• To provide a curated, publicly accessible repository of all 386 surveyed programs and benchmarks to support future research standardization.

Proposed method

• Systematic literature review across five major repositories: ACM Digital Library, IEEE Xplore, Scopus, Google Scholar, and arXiv, retrieving 5,000 publications.
• Identification of 3,749 unique papers through iterative filtering, followed by manual review to select 386 relevant empirical studies on NFP improvement.
• Categorization of selected papers based on: (1) targeted non-functional property, (2) improvement approach (static, evolutionary, exploratory), (3) benchmark type (single software or suite), and (4) software characteristics.
• Keyword-based search and preliminary manual selection of 100 papers to refine search terms and ensure coverage of relevant research areas.
• Aggregation and analysis of benchmark usage patterns, including frequency of reuse and language diversity.
• Public release of all identified programs and benchmarks via a dedicated website: https://bloa.github.io/nfunc_survey/

Experimental results

Research questions

• RQ1 RQ1: How prevalent is empirical work on automated non-functional performance improvement of software, and which non-functional properties are most frequently targeted?
• RQ2 RQ2: What software and benchmarks are most commonly used to validate improvements in non-functional properties?
• RQ3 RQ3: How representative are the benchmarks used in terms of real-world software diversity, including programming languages and application domains?
• RQ4 RQ4: What types of approaches (static, evolutionary, etc.) are most frequently used, and how have they evolved over time?
• RQ5 RQ5: What are the key gaps in current benchmarking practices, particularly regarding multi-objective optimization and support for modern software stacks?

Key findings

• Execution time is the most frequently targeted non-functional property, improved in 62% of the 386 surveyed papers.
• Only 5% of the papers consider multi-objective optimization, indicating a strong focus on single-property improvement despite potential trade-offs.
• Static analysis approaches are dominant (53% of papers), followed by exploratory methods (18% evolutionary, 14% non-evolutionary), with a rising trend in the last decade.
• Only 40% of the 386 papers use benchmark suites; among them, SPEC is the most popular, cited in 33 papers.
• There is a significant mismatch between academic benchmarks and real-world software: Python, JavaScript, and mobile platforms are underrepresented in current research.
• The survey identifies a critical lack of standardized, representative benchmarks for modern software stacks, especially for non-functional properties beyond execution time.