[Paper Review] Toward End-to-End MLOps Tools Map: A Preliminary Study based on a Multivocal Literature Review

Sergio Moreschi, Gilberto Recupito+4 more|arXiv (Cornell University)|Apr 6, 2023

Big Data and Business IntelligenceBusiness, Management and Accounting3 citations

TL;DR

This study conducts a multivocal literature review to map 84 MLOps tools across DevOps phases, identifying their purposes, compatibilities, and incompatibilities. The key contribution is a graphical DevOps map that enables practitioners and researchers to select appropriate tools for each stage of the machine learning lifecycle, supporting end-to-end pipeline design with minimal tool conflicts.

ABSTRACT

MLOps tools enable continuous development of machine learning, following the DevOps process. Different MLOps tools have been presented on the market, however, such a number of tools often create confusion on the most appropriate tool to be used in each DevOps phase. To overcome this issue, we conducted a multivocal literature review mapping 84 MLOps tools identified from 254 Primary Studies, on the DevOps phases, highlighting their purpose, and possible incompatibilities. The result of this work will be helpful to both practitioners and researchers, as a starting point for future investigations on MLOps tools, pipelines, and processes.

Motivation & Objective

• Address the growing confusion among practitioners and researchers in selecting appropriate MLOps tools due to the proliferation of available tools.
• Overcome limitations of prior literature reviews by expanding the scope to include both white and gray literature, and applying snowballing techniques.
• Systematically map MLOps tools to specific DevOps phases to clarify their roles in the machine learning lifecycle.
• Identify potential incompatibilities between tools to guide effective pipeline composition and avoid integration issues.
• Provide a visual, actionable tool map to support both research and industrial adoption of MLOps practices.

Proposed method

• Conducted an internal differentiated replication of a prior multivocal literature review to improve scope and rigor.
• Searched Google and Google Scholar for the first 102 hits, supplemented with snowballing from selected primary studies.
• Collected 254 primary studies (203 gray literature, 51 white literature) and extracted 84 distinct MLOps tools.
• Mapped each tool to one or more DevOps phases (Plan, Code, Build, Test, Release, Deploy, Operate, Monitor) based on its purpose.
• Assessed tool compatibility and incompatibility through cross-referencing tool capabilities and deployment patterns.
• Generated a graphical DevOps map to visually represent tool distribution across phases and support tool selection.

Figure 1: MLOps infinite loop [ 4 ]

Experimental results

Research questions

• RQ1 Which MLOps tools are currently used in practice, and how are they distributed across the DevOps lifecycle phases?
• RQ2 What are the key functional purposes of the identified MLOps tools within each DevOps phase?
• RQ3 Are there any known incompatibilities between MLOps tools that could hinder pipeline integration?
• RQ4 To what extent do end-to-end MLOps platforms support integration with third-party tools?
• RQ5 How can a comprehensive, visual map of MLOps tools enhance the design of end-to-end machine learning pipelines?

Key findings

• A total of 84 distinct MLOps tools were identified from 254 primary studies, with 203 from gray literature and 51 from white literature.
• The majority of end-to-end MLOps platforms do not support integration with external tools, limiting pipeline flexibility.
• No incompatibilities were found among individual tools from different DevOps stages, indicating that they can generally be combined into a cohesive pipeline.
• Most tools are specialized for specific phases, with limited overlap in functionality across stages.
• The study reveals a lack of full automation in current MLOps solutions, with many tools offering overlapping or redundant capabilities.
• The graphical DevOps map provides a clear, actionable overview of tool distribution, aiding both tool selection and pipeline design.

Figure 3: Identified tools and their subcategories to produce MLOps Pipelines