[Paper Review] GPT-4 and Safety Case Generation: An Exploratory Analysis

Mithila Sivakumar, Alvine Boaye Belle+2 more|arXiv (Cornell University)|Dec 9, 2023

Safety Systems Engineering in Autonomy4 citations

TL;DR

This paper investigates GPT-4's capability to generate safety cases using the Goal Structuring Notation (GSN), evaluating its structural correctness, semantic accuracy, and reasonability against ground-truth safety cases for an X-ray system and a tire noise recognition system. GPT-4 demonstrates moderate accuracy and strong semantic alignment with reference cases, suggesting potential for automated safety case assistance, though human oversight remains essential due to hallucination risks and non-determinism.

ABSTRACT

In the ever-evolving landscape of software engineering, the emergence of large language models (LLMs) and conversational interfaces, exemplified by ChatGPT, is nothing short of revolutionary. While their potential is undeniable across various domains, this paper sets out on a captivating expedition to investigate their uncharted territory, the exploration of generating safety cases. In this paper, our primary objective is to delve into the existing knowledge base of GPT-4, focusing specifically on its understanding of the Goal Structuring Notation (GSN), a well-established notation allowing to visually represent safety cases. Subsequently, we perform four distinct experiments with GPT-4. These experiments are designed to assess its capacity for generating safety cases within a defined system and application domain. To measure the performance of GPT-4 in this context, we compare the results it generates with ground-truth safety cases created for an X-ray system system and a Machine-Learning (ML)-enabled component for tire noise recognition (TNR) in a vehicle. This allowed us to gain valuable insights into the model's generative capabilities. Our findings indicate that GPT-4 demonstrates the capacity to produce safety arguments that are moderately accurate and reasonable. Furthermore, it exhibits the capability to generate safety cases that closely align with the semantic content of the reference safety cases used as ground-truths in our experiments.

Motivation & Objective

• To assess GPT-4’s understanding of the Goal Structuring Notation (GSN), a standard for representing safety cases.
• To evaluate GPT-4’s performance in generating safety cases for real-world safety-critical systems, such as an X-ray system and a machine learning-enabled tire noise recognition system.
• To measure the structural correctness, semantic accuracy, and reasonability of GPT-4-generated safety arguments compared to ground-truth safety cases.
• To identify limitations of LLMs in safety case generation, particularly hallucinations and non-deterministic behavior, and to advocate for human-in-the-loop validation.
• To lay the foundation for future automated assessment of safety case quality using extracted GSN rules.

Proposed method

• The authors extracted and formalized the structural and semantic rules of the GSN standard to serve as a knowledge base for evaluation.
• They designed rule-based and generation-based questions to test GPT-4’s understanding of GSN elements and notation.
• Four distinct experiments were conducted, varying in domain knowledge and GSN syntax understanding, to assess GPT-4’s generative capabilities.
• Ground-truth safety cases from two real-world systems (X-ray system and ML-enabled tire noise recognition) were used as reference benchmarks.
• A manual evaluation was performed to assess structural correctness, semantic accuracy, and reasonability of GPT-4 outputs.
• Future work plans to automate evaluation by encoding GSN rules into a verification framework for scalable assessment.

Figure 1. On the right, an example of safety case adapted from (Chouchani et al . , 2022 ) and depicted in the GSN; on the left, the equivalent of the safety case in the structured prose.

Experimental results

Research questions

• RQ1 To what extent can GPT-4 understand and correctly apply the Goal Structuring Notation (GSN) in safety case modeling?
• RQ2 How accurately and reasonably can GPT-4 generate safety cases for safety-critical systems when provided with domain-specific context?
• RQ3 How closely do the semantic contents of GPT-4-generated safety cases align with reference ground-truth safety cases?
• RQ4 What are the key limitations of using GPT-4 for safety case generation, particularly regarding hallucinations and non-deterministic outputs?
• RQ5 Can the integration of GSN rule-based validation improve the reliability of LLM-generated safety cases?

Key findings

• GPT-4 demonstrated a high level of proficiency in understanding GSN, achieving an A-grade performance in comprehension tasks.
• GPT-4-generated safety cases exhibited moderate structural correctness and semantic accuracy, with strong alignment to the semantic content of ground-truth cases.
• The model produced safety arguments that were generally reasonable and contextually relevant, though not fully reliable for autonomous deployment.
• GPT-4 showed non-deterministic behavior, producing varying outputs across multiple runs, which poses challenges for reproducibility.
• Hallucinations were observed, indicating that GPT-4 may generate plausible but factually incorrect safety arguments, necessitating human validation.
• The study concludes that while GPT-4 can assist in safety case generation, human expertise remains essential to ensure safety-critical standards are met.

Figure 2. On the right, the X-ray safety case adapted from (Duan et al . , 2016 ) and depicted in the GSN; on the left, the equivalent of the safety case in the structured prose.