[Paper Review] Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Process

Nadia Nahar, Shurui Zhou+2 more|arXiv (Cornell University)|Oct 19, 2021

Big Data and Business Intelligence20 citations

TL;DR

This paper identifies core collaboration challenges in building ML-enabled systems through interviews with 45 practitioners across 28 organizations. It highlights three key collaboration points—requirements identification, data quality negotiation, and product-model integration—emphasizing communication, documentation, engineering, and process as central themes, and provides actionable recommendations for improving interdisciplinary teamwork in ML development.

ABSTRACT

The introduction of machine learning (ML) components in software projects has created the need for software engineers to collaborate with data scientists and other specialists. While collaboration can always be challenging, ML introduces additional challenges with its exploratory model development process, additional skills and knowledge needed, difficulties testing ML systems, need for continuous evolution and monitoring, and non-traditional quality requirements such as fairness and explainability. Through interviews with 45 practitioners from 28 organizations, we identified key collaboration challenges that teams face when building and deploying ML systems into production. We report on common collaboration points in the development of production ML systems for requirements, data, and integration, as well as corresponding team patterns and challenges. We find that most of these challenges center around communication, documentation, engineering, and process and collect recommendations to address these challenges.

Motivation & Objective

• To understand the collaboration challenges between data scientists and software engineers in building ML-enabled systems for production.
• To identify recurring collaboration points and the associated challenges in requirements, data, and integration across diverse organizational structures.
• To examine how organizational patterns, team composition, and power dynamics influence collaboration effectiveness in ML projects.
• To highlight the underappreciated role of engineering, documentation, and process in ML system development beyond model-centric concerns.
• To provide evidence-based recommendations for improving interdisciplinary collaboration in ML development through better communication, documentation, and planning.

Proposed method

• Conducted semi-structured interviews with 45 practitioners from 28 organizations, including data scientists, software engineers, and managers.
• Analyzed interview data using thematic analysis to identify recurring collaboration points and challenges across organizational contexts.
• Triangulated findings with a literature review on ML systems, software engineering, and collaboration challenges.
• Mapped organizational structures and team patterns using case examples from interviews, including visual representations of team roles and responsibilities.
• Identified common challenges at key collaboration points: requirements decomposition, data negotiation, and integration of ML and software engineering work.
• Developed recommendations based on observed patterns, emphasizing documentation, engineering rigor, and process planning across interdisciplinary teams.

Experimental results

Research questions

• RQ1 What are the primary collaboration points between data scientists and software engineers in ML-enabled system development?
• RQ2 What specific challenges arise at these collaboration points, particularly in requirements, data, and integration?
• RQ3 How do organizational structures, team composition, and power dynamics influence collaboration outcomes in ML projects?
• RQ4 What patterns in team organization and practice are associated with better or worse collaboration outcomes?
• RQ5 What recommendations can be derived to improve collaboration through communication, documentation, engineering, and process?

Key findings

• The three most critical collaboration points are requirements identification, data quality and quantity negotiation, and product-model integration, each posing distinct interdisciplinary challenges.
• Communication breakdowns are pervasive, especially due to differing technical backgrounds and terminology, with many practitioners reporting misalignment between data scientists and software engineers.
• Documentation is often inadequate or inconsistent, with no standardized practices for specifying model requirements, data expectations, or model qualities like fairness and explainability.
• Engineering practices are frequently underestimated; teams struggle with productionizing models due to lack of automation, testing, and monitoring, despite the increased complexity ML introduces.
• Process challenges stem from ML’s non-linear, exploratory nature clashing with traditional software development lifecycles, leading to misaligned timelines and unclear responsibilities.
• Organizations with better collaboration outcomes often invest in interdisciplinary training, clear interface documentation, and early involvement of both data science and software engineering teams in planning.