[Paper Review] Towards an MLOps Architecture for XAI in Industrial Applications

Leonhard Faubel, Thomas Woudsma+9 more|arXiv (Cornell University)|Sep 22, 2023

Big Data and Business Intelligence5 citations

TL;DR

This paper proposes a novel MLOps architecture integrating Explainable AI (XAI) into industrial machine learning workflows, enabling end-to-end explainability, feedback collection, and model management. The architecture, implemented in the EXPLAIN project across industrial use cases, uses standardized tools like MLflow, Seldon Core, and Evidently to streamline model lifecycle management with built-in XAI and feedback mechanisms, enhancing trust and transparency for non-technical stakeholders.

ABSTRACT

Machine learning (ML) has become a popular tool in the industrial sector as it helps to improve operations, increase efficiency, and reduce costs. However, deploying and managing ML models in production environments can be complex. This is where Machine Learning Operations (MLOps) comes in. MLOps aims to streamline this deployment and management process. One of the remaining MLOps challenges is the need for explanations. These explanations are essential for understanding how ML models reason, which is key to trust and acceptance. Better identification of errors and improved model accuracy are only two resulting advantages. An often neglected fact is that deployed models are bypassed in practice when accuracy and especially explainability do not meet user expectations. We developed a novel MLOps software architecture to address the challenge of integrating explanations and feedback capabilities into the ML development and deployment processes. In the project EXPLAIN, our architecture is implemented in a series of industrial use cases. The proposed MLOps software architecture has several advantages. It provides an efficient way to manage ML models in production environments. Further, it allows for integrating explanations into the development and deployment processes.

Motivation & Objective

• Address the gap in industrial MLOps solutions that lack native support for explainability and stakeholder feedback.
• Enable domain experts without ML expertise to actively participate in model development, deployment, and validation.
• Develop a scalable, modular MLOps stack that natively integrates XAI tools and feedback loops for industrial applications.
• Ensure model reliability through continuous monitoring of data drift, concept drift, and prediction quality.
• Bridge the divide between technical ML teams and industrial practitioners by making model decisions transparent and interactive.

Proposed method

• Design a modular MLOps architecture based on industrial project requirements and real-world deployment constraints.
• Integrate XAI tools such as H2O, IBM AI Explainability 360, and custom explainers into the model training and serving pipeline.
• Use MLflow for tracking experiments, versioning models and explanations, and managing the model registry.
• Deploy models via Seldon Core on Kubernetes for scalable, production-grade inference with explainability support.
• Implement data monitoring with Evidently and Grafana to detect data drift and concept drift in real time.
• Develop a prototype feedback GUI (Django-based) for labeling and correcting predictions, with plans to integrate advanced image-specific explainers.

Figure 1: Adopted explain life cycle from [ 14 ] .

Experimental results

Research questions

• RQ1 How can MLOps be extended to natively support explainability and feedback in industrial machine learning deployments?
• RQ2 What architectural components are required to enable non-technical domain experts to understand and contribute to ML model development and inference?
• RQ3 How can data drift and concept drift be detected and managed within an integrated MLOps and XAI pipeline?
• RQ4 What role do standardized tools like MLflow and Seldon Core play in enabling scalable, versioned, and explainable model management?
• RQ5 How can feedback from domain experts be systematically captured and integrated into model improvement cycles?

Key findings

• The proposed MLOps architecture successfully integrates XAI into the full ML lifecycle, enabling transparency and stakeholder participation in industrial settings.
• The use of MLflow for model and explanation versioning allows for reproducibility and auditability of model decisions.
• Data and concept drift are effectively monitored using Evidently and Grafana, supporting proactive model maintenance.
• A prototype feedback GUI for image data labeling and correction has been implemented, with plans to extend it with state-of-the-art image explainers.
• Model serving via Seldon Core enables horizontal scalability and integration of explainability at inference time.
• Despite progress, automated retraining, alerting, and full feedback loop integration remain underdeveloped and are identified as key areas for future work.

Figure 2: EXPLAIN MLOps Software Architecture in five domains and eight components. Major XAI functionality in bold , but it is not limited to only those components.