[Paper Review] Software Engineering for AI-Based Systems: A Survey

Silverio Martínez‐Fernández, Justus Bogner+6 more|arXiv (Cornell University)|May 5, 2021

Software Engineering ResearchComputer Science238 references291 citations

TL;DR

This paper presents a systematic mapping study of software engineering for AI-based systems (SE4AI) from 2010 to 2020, analyzing 248 studies to identify key SE approaches, challenges, and research trends. It reveals that software testing and quality are the most studied areas, while maintenance remains under-researched, and data-related issues are the dominant challenges, highlighting the need for updated SE practices in AI systems.

ABSTRACT

AI-based systems are software systems with functionalities enabled by at least one AI component (e.g., for image- and speech-recognition, and autonomous driving). AI-based systems are becoming pervasive in society due to advances in AI. However, there is limited synthesized knowledge on Software Engineering (SE) approaches for building, operating, and maintaining AI-based systems. To collect and analyze state-of-the-art knowledge about SE for AI-based systems, we conducted a systematic mapping study. We considered 248 studies published between January 2010 and March 2020. SE for AI-based systems is an emerging research area, where more than 2/3 of the studies have been published since 2018. The most studied properties of AI-based systems are dependability and safety. We identified multiple SE approaches for AI-based systems, which we classified according to the SWEBOK areas. Studies related to software testing and software quality are very prevalent, while areas like software maintenance seem neglected. Data-related issues are the most recurrent challenges. Our results are valuable for: researchers, to quickly understand the state of the art and learn which topics need more research; practitioners, to learn about the approaches and challenges that SE entails for AI-based systems; and, educators, to bridge the gap among SE and AI in their curricula.

Motivation & Objective

• To synthesize the current state of the art in software engineering for AI-based systems (SE4AI) to support researchers, practitioners, and educators.
• To identify the most prevalent SE approaches, challenges, and research trends in AI system development, operation, and maintenance.
• To analyze the distribution of research across SWEBOK knowledge areas and highlight under-explored domains such as software maintenance.
• To provide a taxonomy and conceptual framework for SE4AI to improve clarity and consistency in future research.
• To identify key challenges, especially data-related ones, and recommend improvements in SE standards and practices for AI systems.

Proposed method

• Conducted a systematic mapping study (SMS) following the guidelines of the systematic literature review process.
• Searched 12 databases and arXiv for studies published between January 2010 and March 2020, applying predefined inclusion/exclusion criteria.
• Used a three-stage screening process: title/abstract, full-text, and final eligibility, with dual independent screening and consensus resolution.
• Classified studies using the SWEBOK knowledge areas (e.g., software testing, quality, requirements) to map SE approaches.
• Identified challenges by analyzing study contributions and grouping them into themes, especially focusing on data, ethics, and system-specific issues.
• Performed qualitative analysis and iterative coding with the research team to refine categories and ensure consistency.

Experimental results

Research questions

• RQ1 What are the main software engineering approaches used in the development and maintenance of AI-based systems, and how are they distributed across SWEBOK knowledge areas?
• RQ2 Which quality attributes and application domains are most frequently targeted in SE4AI research, and what are the dominant AI technologies used?
• RQ3 What are the most prominent challenges in engineering AI-based systems, and how do they relate to specific SE knowledge areas?
• RQ4 How has the research landscape in SE4AI evolved from 2010 to 2020, and what gaps exist in current research, particularly in underrepresented areas like software maintenance?
• RQ5 To what extent do primary studies address practical challenges such as data quality, model robustness, and ethical considerations in AI systems?

Key findings

• More than two-thirds of SE4AI studies were published after 2018, indicating rapid growth in the field.
• Software testing (115 studies) and software quality (59 studies) are the most researched SE areas, with a strong focus on test case generation and adaptation of standards like ISO 26262.
• Data-related issues are the most recurrent challenges, cited in 25% of identified challenges, with data quality, bias, and scarcity being key concerns.
• Software maintenance is significantly under-represented, with fewer than 10 studies identified, suggesting a major research gap.
• The automotive domain is the most frequent application area, while nearly half of the studies do not specify a particular domain.
• Deep learning is the most explicitly mentioned AI technique, used in almost all contributions, reflecting its dominance in current AI-based systems.