[Paper Review] Leveraging Large Language Models (LLMs) for Process Mining (Technical Report)

Alessandro Berti, Mahnaz Sadat Qafari|arXiv (Cornell University)|Jul 24, 2023

Business Process Modeling and Analysis8 citations

TL;DR

The paper investigates using large language models (GPT-4 and Bard) to analyze process mining artifacts by converting logs and models into textual abstractions and applying various prompting strategies.

ABSTRACT

This technical report describes the intersection of process mining and large language models (LLMs), specifically focusing on the abstraction of traditional and object-centric process mining artifacts into textual format. We introduce and explore various prompting strategies: direct answering, where the large language model directly addresses user queries; multi-prompt answering, which allows the model to incrementally build on the knowledge obtained through a series of prompts; and the generation of database queries, facilitating the validation of hypotheses against the original event log. Our assessment considers two large language models, GPT-4 and Google's Bard, under various contextual scenarios across all prompting strategies. Results indicate that these models exhibit a robust understanding of key process mining abstractions, with notable proficiency in interpreting both declarative and procedural process models. In addition, we find that both models demonstrate strong performance in the object-centric setting, which could significantly propel the advancement of the object-centric process mining discipline. Additionally, these models display a noteworthy capacity to evaluate various concepts of fairness in process mining. This opens the door to more rapid and efficient assessments of the fairness of process mining event logs, which has significant implications for the field. The integration of these large language models into process mining applications may open new avenues for exploration, innovation, and insight generation in the field.

Motivation & Objective

• Motivate the integration of large language models with traditional and object-centric process mining artifacts.
• Develop textual abstractions for process mining artifacts to enable LLM comprehension.
• Evaluate prompting strategies (direct, multi-prompt, and database-query generation) with LLMs.
• Assess performance of LLMs on procedural and declarative models and fairness concepts in process mining.

Proposed method

• Create textual encodings of process mining artifacts (DFG, Petri nets, OC-DFG, DECLARE, temporal profiles).
• Describe and implement prompting strategies: direct answering, multi-prompt answering, and generation of database queries to validate hypotheses against event logs.
• Experiment with two LLMs (GPT-4 and Google Bard) across different contexts and artifact types.
• Discuss preprocessing and abbreviation strategies to fit context window constraints for complex models (e.g., DECLARE models).
• Outline feature extraction approaches to convert event logs into numeric inputs for ML tasks in process mining (one-hot encoding, aggregates, sequences, n-grams, embeddings).
• Address fairness concepts in process mining, including individual, group, procedural, and counterfactual fairness, and discuss public data availability for fairness evaluation.

Experimental results

Research questions

• RQ1 How effectively can LLMs understand and reason about traditional and object-centric process mining artifacts when encoded as text?
• RQ2 What prompting strategies enable robust analysis and hypothesis validation for process mining tasks using LLMs?
• RQ3 To what extent can LLMs support procedural, declarative, and object-centric process models, and assess fairness in event logs?

Key findings

• LLMs (GPT-4 and Bard) show robust understanding of key process mining abstractions and can interpret both declarative and procedural models.
• LLMs perform well in object-centric settings, supporting advancement of object-centric process mining.
• Textual abstractions enable LLMs to reason about directly-follows graphs, Petri nets, and DECLARE/temporal models.
• Prompting strategies including multi-prompt and database-query generation aid in hypothesis validation against original event logs.
• LLMs demonstrate capacity to evaluate fairness concepts in process mining, suggesting potential for rapid fairness assessments.