[Paper Review] How Generative AI models such as ChatGPT can be (Mis)Used in SPC Practice, Education, and Research? An Exploratory Study

Fadel M. Megahed, Ying‐Ju Chen+3 more|arXiv (Cornell University)|Feb 17, 2023

Business Process Modeling and AnalysisBusiness, Management and Accounting32 citations

TL;DR

The paper evaluates ChatGPT’s capabilities in SPC practice, learning, and research, highlighting benefits for efficiency and potential misuses, with findings that it handles structured tasks well but struggles with nuanced terms and scratch coding, underlining the need for validation and supplementary methods.

ABSTRACT

Generative Artificial Intelligence (AI) models such as OpenAI's ChatGPT have the potential to revolutionize Statistical Process Control (SPC) practice, learning, and research. However, these tools are in the early stages of development and can be easily misused or misunderstood. In this paper, we give an overview of the development of Generative AI. Specifically, we explore ChatGPT's ability to provide code, explain basic concepts, and create knowledge related to SPC practice, learning, and research. By investigating responses to structured prompts, we highlight the benefits and limitations of the results. Our study indicates that the current version of ChatGPT performs well for structured tasks, such as translating code from one language to another and explaining well-known concepts but struggles with more nuanced tasks, such as explaining less widely known terms and creating code from scratch. We find that using new AI tools may help practitioners, educators, and researchers to be more efficient and productive. However, in their current stages of development, some results are misleading and wrong. Overall, the use of generative AI models in SPC must be properly validated and used in conjunction with other methods to ensure accurate results.

Motivation & Objective

• Assess ChatGPT’s ability to generate SPC-related code for practice tasks.
• Evaluate ChatGPT’s explanations of SPC concepts in learning contexts.
• Investigate ChatGPT’s ability to create SPC knowledge assets such as frameworks, syllabi, and research issue identification.

Proposed method

• Evaluate prompts that request R code for control charts and task translation between languages.
• Compare ChatGPT outputs to textbooks and literature to assess accuracy of explanations.
• Test ChatGPT’s ability to create SPC frameworks, course syllabi, and identify open research questions.
• Attempt to run and debug ChatGPT-generated code in R and Python and document execution outcomes.
• Provide qualitative assessments of what worked and what failed across practice, learning, and research in SPC.

(a) A representative sample of LLM models released/published in the last three years.

Experimental results

Research questions

• RQ1 Can ChatGPT generate correct and executable code for SPC tasks (e.g., X-bar charts) in R and Python?
• RQ2 How accurate are ChatGPT’s explanations of SPC concepts such as Phase 1 vs Phase 2, zero-state ARL, and univariate/multivariate/profile monitoring?
• RQ3 To what extent can ChatGPT create usable SPC knowledge assets, such as frameworks, DMAIC templates, and open research issues?
• RQ4 What are the limitations and potential misuses of ChatGPT in SPC practice, learning, and research?

Key findings

• ChatGPT can provide code and explanations for structured SPC tasks but often mislabels code blocks, has parameter and function argument errors, and may produce incorrect charts without debugging.
• ChatGPT’s explanations of Phase 1 vs Phase 2, zero-state ARL, and monitoring approaches contain inaccuracies or incomplete details that require expert validation.
• ChatGPT can generate knowledge assets like a seven-step MSP framework and DMAIC templates, but these outputs may lack sufficient statistical detail and need tailoring by a practitioner.
• Alternative code approaches (tidyverse vs base R) and cross-language translation to Python show mixed success, with some outputs needing substantial corrections to be correct.
• The study demonstrates that AI tools can increase efficiency but also risk producing misleading or wrong results, underscoring the need for validation and augmentation with traditional methods.

(b) The unprecedented adoption of ChatGPT compared to popular technology platforms.