[Paper Review] Software engineering for artificial intelligence and machine learning software: A systematic literature review

Elizamary Nascimento, Anh Nguyen‐Duc+2 more|arXiv (Cornell University)|Nov 7, 2020

Software Engineering Research62 references41 citations

TL;DR

This paper surveys how software engineering is applied to AI/ML systems, identifying practices, contexts, and key challenges from 1990–2019.

ABSTRACT

Artificial Intelligence (AI) or Machine Learning (ML) systems have been widely adopted as value propositions by companies in all industries in order to create or extend the services and products they offer. However, developing AI/ML systems has presented several engineering problems that are different from those that arise in, non-AI/ML software development. This study aims to investigate how software engineering (SE) has been applied in the development of AI/ML systems and identify challenges and practices that are applicable and determine whether they meet the needs of professionals. Also, we assessed whether these SE practices apply to different contexts, and in which areas they may be applicable. We conducted a systematic review of literature from 1990 to 2019 to (i) understand and summarize the current state of the art in this field and (ii) analyze its limitations and open challenges that will drive future research. Our results show these systems are developed on a lab context or a large company and followed a research-driven development process. The main challenges faced by professionals are in areas of testing, AI software quality, and data management. The contribution types of most of the proposed SE practices are guidelines, lessons learned, and tools.

Motivation & Objective

• Understand the current state of the art in applying software engineering to AI/ML systems.
• Identify SE practices, guidelines, lessons learned, and tools proposed for AI/ML software.
• Analyze contexts in which SE practices are applied (lab vs. industry) and assess applicability.
• Highlight limitations and open challenges to guide future research.

Proposed method

• Conduct a systematic literature review of AI/ML SE studies published from 1990 to 2019.
• Classify and summarize SE practices as guidelines, lessons learned, or tools.
• Synthesize context and applicability of SE practices across settings.
• Identify limitations and open research challenges to steer future work.

Experimental results

Research questions

• RQ1 What SE practices have been proposed for AI/ML software development?
• RQ2 In which contexts (lab, industry, etc.) are these SE practices applied and how applicable are they?
• RQ3 What are the main challenges and limitations reported in applying SE to AI/ML systems?
• RQ4 What types of contributions (guidelines, lessons learned, tools) dominate the literature?
• RQ5 What gaps exist that future research should address?

Key findings

• AI/ML systems are often developed in a lab context or within large companies.
• Development tends to follow a research-driven process.
• Primary challenges include testing, AI software quality, and data management.
• Most proposed SE practices are guidelines, lessons learned, or tools.