[Paper Review] Evaluating Explainable Methods for Predictive Process Analytics: A Functionally-Grounded Approach

Mythreyi Velmurugan, Chun Ouyang+2 more|arXiv (Cornell University)|Dec 8, 2020

Business Process Modeling and Analysis4 citations

TL;DR

This paper proposes functionally-grounded evaluation metrics—fidelity and stability—to assess explainable AI methods (LIME and SHAP) in predictive process analytics. Using XGBoost models on three real-world event logs, it finds that both methods show comparable fidelity but low overall faithfulness, especially under single bucketing and aggregate encoding, with no consistent link between prefix length and explanation quality, highlighting challenges in practical deployment despite strong black-box accuracy.

ABSTRACT

Predictive process analytics focuses on predicting the future states of running instances of a business process. While advanced machine learning techniques have been used to increase accuracy of predictions, the resulting predictive models lack transparency. Current explainable machine learning methods, such as LIME and SHAP, can be used to interpret black box models. However, it is unclear how fit for purpose these methods are in explaining process predictive models. In this paper, we draw on evaluation measures used in the field of explainable AI and propose functionally-grounded evaluation metrics for assessing explainable methods in predictive process analytics. We apply the proposed metrics to evaluate the performance of LIME and SHAP in interpreting process predictive models built on XGBoost, which has been shown to be relatively accurate in process predictions. We conduct the evaluation using three open source, real-world event logs and analyse the evaluation results to derive insights. The research contributes to understanding the trustworthiness of explainable methods for predictive process analytics as a fundamental and key step towards human user-oriented evaluation.

Motivation & Objective

• To address the lack of systematic evaluation of explainable AI methods in predictive process analytics.
• To propose functionally-grounded metrics—fidelity and stability—for assessing explanation quality independent of human evaluation.
• To evaluate the performance of LIME and SHAP in explaining XGBoost-based predictive models across diverse real-world event logs.
• To identify how data encoding, bucketing strategies, and prefix length affect explanation quality.
• To provide insights into the trustworthiness and practical limitations of explainable methods in process prediction contexts.

Proposed method

• Proposes functionally-grounded evaluation metrics: explanation fidelity (how well the explanation reflects the black-box model) and stability (consistency of feature importance across perturbations).
• Applies LIME and SHAP as post-hoc explanation methods to interpret XGBoost models trained on three real-world event logs: Production, Sepsis Cases, and BPIC2012.
• Uses multiple preprocessing configurations: single bucketing vs. prefix-length bucketing, and aggregate vs. index-based encoding.
• Measures fidelity by comparing the prediction of the surrogate model (LIME/SHAP) to the original black-box model on perturbed instances.
• Measures stability by computing the similarity of feature importance rankings across multiple perturbations of the same instance.
• Employs a systematic experimental design across varying prefix lengths and encoding methods to analyze trends in explanation quality.

Experimental results

Research questions

• RQ1 How do LIME and SHAP perform in terms of fidelity when explaining XGBoost models on real-world process prediction tasks?
• RQ2 What is the impact of different encoding and bucketing strategies on the stability and fidelity of explanations?
• RQ3 Is there a consistent relationship between prefix length and explanation quality in terms of fidelity or stability?
• RQ4 How do the characteristics of the dataset (e.g., size, sparsity) influence explanation quality?
• RQ5 To what extent do explanation methods remain reliable when the black-box model is highly accurate but complex?

Key findings

• LIME and SHAP show comparable fidelity across most configurations, but overall faithfulness remains low, especially when single bucketing and aggregate encoding are used together.
• The combination of single bucketing and aggregate encoding, while maximizing black-box accuracy, results in the least faithful explanations, indicating a trade-off between accuracy and explainability.
• No consistent trend between prefix length and fidelity is observed, except in the BPIC2012 dataset, where longer prefixes correlate with higher fidelity, likely due to better black-box performance at larger prefix lengths.
• Stability and fidelity show no direct correlation; poor stability does not necessarily imply poor fidelity, suggesting that ensemble-based explanation selection may mitigate instability.
• In the Sepsis Cases dataset with index-based encoding, explanations are more uniformly faithful near the decision boundary (prediction probability ≈ 0.5), but less precise when predictions are more confident, likely due to model underfitting and data sparsity.
• The results suggest that even when black-box models are accurate, explanation methods may not reliably reflect their behavior, limiting practical usability without further refinement or method selection.