[Paper Review] Chit-Chat or Deep Talk: Prompt Engineering for Process Mining

Urszula Jessen, Michal Sroka+1 more|arXiv (Cornell University)|Jul 19, 2023

Business Process Modeling and Analysis8 citations

TL;DR

The paper presents a framework that uses large language models with role-specific prompts and an orchestrator to enable conversational querying of process mining data, and evaluates its effectiveness on the BPI Challenge 2019 dataset with GPT-3.5-turbo and GPT-4.

ABSTRACT

This research investigates the application of Large Language Models (LLMs) to augment conversational agents in process mining, aiming to tackle its inherent complexity and diverse skill requirements. While LLM advancements present novel opportunities for conversational process mining, generating efficient outputs is still a hurdle. We propose an innovative approach that amend many issues in existing solutions, informed by prior research on Natural Language Processing (NLP) for conversational agents. Leveraging LLMs, our framework improves both accessibility and agent performance, as demonstrated by experiments on public question and data sets. Our research sets the stage for future explorations into LLMs' role in process mining and concludes with propositions for enhancing LLM memory, implementing real-time user testing, and examining diverse data sets.

Motivation & Objective

• Address the usability gap in process mining by enabling non-experts to query event data via conversational agents.
• Develop an architecture that uses LLMs to emulate diverse process mining roles (analyst, data engineer, domain expert) and orchestrate prompts.
• Evaluate the effectiveness of the proposed approach on real questions and datasets to identify strengths and limitations.

Proposed method

• Proposes a generic prompt engineering framework that translates user questions into SQL queries against process event logs.
• Introduces an orchestrator that reuses similar past questions via embeddings to skip re-computation when possible.
• Promotes a multi-prompt, role-based prompt strategy and a prompting workflow that includes feedback from SQL execution errors to refine queries.
• Evaluates with GPT-3.5-turbo and GPT-4, using the BPI Challenge 2019 dataset and a corpus of questions from Barbieri et al.
• Assesses answers along dimensions of fully/partially correct and understood/partially understood, using manual evaluation criteria.

Figure 1: Architecture of a conversational agent for process mining

Experimental results

Research questions

• RQ1 Can LLMs understand and answer process mining questions via a structured, prompt-driven conversational interface?
• RQ2 Does orchestrating role-specific prompts and iterative SQL queries improve accuracy and usefulness of answers for process mining queries?
• RQ3 What is the trade-off between model type (GPT-3.5-turbo vs GPT-4) and cost versus accuracy in this setting?
• RQ4 How does embedding-based retrieval of similar questions impact response time and success rate?

Key findings

• In 77% of instances, LLMs fully or partially comprehended the question and outlined an appropriate solution.
• In 68% of cases, the model provided either the correct or a partially correct answer.
• Table 1 (comparison of results): Answered 285 (36%), Partially answered 254 (32%), Understood 155 (19%), Partially understood 459 (58%).
• GPT-4 with few-shot learning improved additional cases beyond GPT-3.5-Turbo, though cost is higher; zero-shot with GPT-3.5-Turbo yielded 61 cases (partially/fully) improved after error-driven corrections, and GPT-4 added 193 partially and 178 fully answered cases in few-shot mode.
• Table 2 shows zero-shot and few-shot performance by model: GPT-3.5 Zero Shot 49 partial, Few Shot 12; GPT-4 Zero Shot 0, Few Shot 193; overall better with GPT-4 in few-shot mode.
• The study highlights trade-offs between accuracy and cost, with overall practical costs around $60 for ~800 questions.

Figure 2: The general process of prompt engineering.