[Paper Review] MLOps Challenges in Multi-Organization Setup: Experiences from Two Real-World Cases

Tuomas Granlund, Aleksi Kopponen+3 more|arXiv (Cornell University)|Mar 16, 2021

Big Data and Business Intelligence19 references39 citations

TL;DR

The paper presents two real-world multi-organization MLOps cases (Oravizio and AuroraAI), analyzing integration, data ownership, modeling, monitoring, and governance challenges, and proposes patterns and organizational practices to address them.

ABSTRACT

The emerging age of connected, digital world means that there are tons of data, distributed to various organizations and their databases. Since this data can be confidential in nature, it cannot always be openly shared in seek of artificial intelligence (AI) and machine learning (ML) solutions. Instead, we need integration mechanisms, analogous to integration patterns in information systems, to create multi-organization AI/ML systems. In this paper, we present two real-world cases. First, we study integration between two organizations in detail. Second, we address scaling of AI/ML to multi-organization context. The setup we assume is that of continuous deployment, often referred to DevOps in software development. When also ML components are deployed in a similar fashion, term MLOps is used. Towards the end of the paper, we list the main observations and draw some final conclusions. Finally, we propose some directions for future work.

Motivation & Objective

• Motivate the study of MLOps in multi-organization contexts where data cannot be freely shared.
• Identify and analyze integration and scaling challenges across organizational boundaries.
• Provide practical patterns and governance concepts to enable multi-organization ML deployment.
• Highlight how data ownership and regulatory constraints shape MLOps practices.

Proposed method

• Present two real-world case studies: Oravizio (two organizations sharing a medical ML service) and AuroraAI (multi-organization scaling of AI for public services).
• Analyze data set ownership, model sharing, monitoring, and governance in each case.
• Discuss patterns and architectural decisions used to enable cross-organization ML artifacts (e.g., treating models as shared software components).
• Relate findings to CD4ML and DevOps/MLOps concepts to frame continuous deployment in ML contexts.

Experimental results

Research questions

• RQ1 How can ML features be integrated across organizational boundaries while preserving data privacy and ownership constraints?
• RQ2 What practical patterns and organizational practices enable reliable multi-organization MLOps?
• RQ3 How do data governance, regulatory constraints, and model handovers affect ML deployment across organizations?
• RQ4 What considerations are essential when scaling ML solutions to multi-organization ecosystems like AuroraAI?

Key findings

• Data ownership and regulatory constraints make data movement across organizations difficult, driving data lake and sharing patterns that preserve boundaries.
• A shared ML model can serve as the primary artifact for cross-organization collaboration when data cannot be exposed, with model development and deployment distributed across organizations.
• Monitoring in multi-organization MLOps must account for biases, drift, and governance, often requiring human-in-the-loop decisions and clear responsibility boundaries.
• Patterns such as Evaluator (Oravizio) and User Delegation (AuroraAI) illustrate practical approaches to inter-organizational ML collaboration.
• Successful multi-organization ML requires joint operating modes (OrgOps) and collaboration beyond single-organizational DevOps, recognizing cultural and governance differences.