[Paper Review] Search-Based Software Engineering for Self-Adaptive Systems: Survey, Disappointments, Suggestions and Opportunities

Tao Chen, Miqing Li+2 more|arXiv (Cornell University)|Jan 22, 2020

Software Engineering Research4 citations

TL;DR

This paper presents the first comprehensive survey of Search-Based Software Engineering (SBSE) for Self-Adaptive Systems (SASs), analyzing 74 primary studies from 2009–2019. It identifies five critical disappointments—unjustified algorithm choices, limited multi-objective formulations, questionable evaluation methods, lack of algorithm specialization, and weak generalization—while proposing eight under-explored research opportunities to advance the field.

ABSTRACT

Search-Based Software Engineering (SBSE) is a promising paradigm that exploits the computational search to optimize different processes when engineering complex software systems. Self-adaptive system (SAS) is one category of such complex systems that permits to optimize different functional and non-functional objectives/criteria under changing environments (e.g., requirements and workload), which involves problems that are subject to search. In this regard, over years, there has been a considerable amount of work that investigates SBSE for SASs. In this paper, we provide the first systematic and comprehensive survey exclusively on SBSE for SASs, covering papers in 27 venues from 7 repositories, which eventually leads to several key statistics from the most notable 74 primary studies in this particular field of research. Our results, surprisingly, have revealed five disappointments that are of utmost importance and can result in serve consequences but have been overwhelmingly ignored in existing studies. We provide theoretical and/or experimental evidence to justify our arguments against the disappointments, present suggestions, and highlight the promising research opportunities towards their mitigation. We also elaborate on three other emergent, but currently under-explored opportunities for future work on SBSE for SASs. By mitigating the disappointments revealed in this work, together with the highlighted opportunities, we hope to be able to excite a much more significant growth in this particular research direction.

Motivation & Objective

• To address the lack of a systematic overview of SBSE for self-adaptive systems (SASs), which has hindered field growth.
• To identify and analyze recurring methodological shortcomings—termed 'disappointments'—in current SBSE research for SASs.
• To provide evidence-based suggestions for improving research validity and rigor in SBSE for SASs.
• To highlight under-explored research opportunities that could significantly advance the field of SBSE for SASs.
• To bridge the gap between SBSE and SAS communities by offering actionable insights for practitioners and researchers.

Proposed method

• Conducted a systematic literature survey across 27 venues and 7 repositories, identifying 409 papers for review and selecting 74 primary studies for in-depth analysis.
• Applied a multi-level justification framework (L1–L4) to evaluate the rationale behind search algorithm choices and multi-objective formulations.
• Classified studies based on search type (single vs. multi-objective), algorithm usage, evaluation methods, and subject SAS characteristics.
• Assessed the use of quality indicators (e.g., hypervolume, generational distance) in Pareto optimization and evaluated their justifications.
• Analyzed the representation, fitness function design, and simulator usage in SBSE for SASs to assess problem-specific customization.
• Identified research gaps through thematic and statistical analysis of trends across time, domains, and algorithmic choices.

Experimental results

Research questions

• RQ1 What are the most commonly used search algorithms in SBSE for SASs, and how justified are their choices in the context of SAS problems?
• RQ2 How are multi-objective optimization problems formulated in SBSE for SASs, and what are the limitations in their formulation and evaluation?
• RQ3 What evaluation methods are used for Pareto search in SBSE for SASs, and how rigorous are their justifications?
• RQ4 To what extent are search algorithms specialized for SAS-specific characteristics, and how much is their internal design modified?
• RQ5 How generalizable are the results across different SASs, and what is the diversity of subject systems used in current studies?

Key findings

• NSGA-II is the dominant algorithm for Pareto search, while GA, LS, and IP solvers are most common for single or aggregated objectives, often without sufficient justification.
• Over 50% of studies using Pareto search do not employ any generic quality indicator, and among those that do, justifications are typically at L2 or L3, indicating weak theoretical grounding.
• Only 15% of studies provide L4-level justification for algorithm selection, with most relying on L1 or L2 justifications such as 'commonly used' or 'available in libraries'.
• There is a growing disconnect between problem nature and SE/SAS expertise, with most studies focusing narrowly on representation and fitness function rather than system-specific adaptation logic.
• Simulators are the most common subject systems, and the majority of studies evaluate on only one SAS per study, limiting generalization potential.
• The field exhibits a strong bias toward latency and cost as primary objectives, with limited exploration of other non-functional criteria such as energy efficiency or security.