[Paper Review] KRAIL: A Knowledge-Driven Framework for Base Human Reliability Analysis Integrating IDHEAS and Large Language Models

Xingyu Xiao, Peng Chen+5 more|arXiv (Cornell University)|Dec 20, 2024

Risk and Safety AnalysisDecision Sciences3 citations

TL;DR

This paper proposes KRAIL, a knowledge-driven, two-stage framework that integrates IDHEAS-DATA with large language models (LLMs) and knowledge graphs to enable semi-automated base human error probability (BHEP) estimation. By leveraging multi-agent task decomposition and retrieval-augmented generation via a Neo4j knowledge graph, KRAIL reduces BHEP estimation time to under 150 seconds while maintaining high accuracy, offering a scalable, expert-informed alternative to manual HRA methods.

ABSTRACT

Human reliability analysis (HRA) is crucial for evaluating and improving the safety of complex systems. Recent efforts have focused on estimating human error probability (HEP), but existing methods often rely heavily on expert knowledge,which can be subjective and time-consuming. Inspired by the success of large language models (LLMs) in natural language processing, this paper introduces a novel two-stage framework for knowledge-driven reliability analysis, integrating IDHEAS and LLMs (KRAIL). This innovative framework enables the semi-automated computation of base HEP values. Additionally, knowledge graphs are utilized as a form of retrieval-augmented generation (RAG) for enhancing the framework' s capability to retrieve and process relevant data efficiently. Experiments are systematically conducted and evaluated on authoritative datasets of human reliability. The experimental results of the proposed methodology demonstrate its superior performance on base HEP estimation under partial information for reliability assessment.

Motivation & Objective

• To address the time-consuming and expert-dependent nature of traditional base human error probability (BHEP) estimation in human reliability analysis (HRA).
• To reduce reliance on manual, subjective expert judgment in BHEP computation by introducing an automated, LLM-powered framework.
• To enhance the efficiency and accuracy of BHEP estimation through knowledge graph-augmented retrieval and multi-agent task decomposition.
• To provide a scalable, user-friendly solution for HRA practitioners, especially in high-risk industries like nuclear energy and aviation.
• To validate the framework’s performance using authoritative HRA datasets and real-world case studies.

Proposed method

• The framework employs a two-stage process: first, a multi-agent system decomposes human tasks into cognitive components, context, time constraints, and error measures.
• Second, the LLM processes natural language representations derived from an IDHEAS-DATA-powered knowledge graph to infer missing parameters for BHEP calculation.
• A Neo4j-based knowledge graph stores and retrieves domain-specific HRA data, enabling structured, context-aware reasoning for error rate prediction.
• Retrieval-augmented generation (RAG) techniques are used to ground LLM outputs in factual IDHEAS-DATA, improving reliability and reducing hallucination.
• A Gradio-based web interface enables end-users to input case data, trigger task decomposition, and generate BHEP estimates with expert review capabilities.
• The system supports iterative refinement, allowing experts to validate and adjust parameters before final error rate computation.

Figure 1: IDHEAS-ECA HRA Process [ 3 ]

Experimental results

Research questions

• RQ1 Can an LLM-based framework reduce the time required for base human error probability (BHEP) estimation while maintaining accuracy compared to manual methods?
• RQ2 How effectively can a knowledge graph enhance retrieval and reasoning for HRA parameter estimation in low-information scenarios?
• RQ3 To what extent does multi-agent task decomposition improve the consistency and completeness of human performance analysis in HRA?
• RQ4 Can the integration of IDHEAS-DATA with LLMs produce BHEP estimates that align with expert-validated benchmarks?
• RQ5 How does the RAG-enhanced LLM approach compare to standard LLM inference in terms of factual consistency and reliability for HRA applications?

Key findings

• The KRAIL framework reduces base HEP estimation time to under 150 seconds, significantly improving efficiency over manual methods.
• The framework achieves high accuracy in BHEP estimation, even under partial information, demonstrating robustness in low-data scenarios.
• The integration of knowledge graphs with LLMs enhances retrieval precision, reducing hallucination and improving factual grounding of generated parameters.
• Ablation studies confirm that both the multi-agent decomposition and knowledge graph components are essential for optimal performance.
• The web-based interface enables practical, interactive use by domain experts, supporting iterative refinement and expert validation of results.
• The framework demonstrates strong generalization across diverse HRA case studies, indicating broad applicability in high-risk industries.

Figure 2: KRAIL: A Knowledge-Driven Framework Integrating IDHEAS and Large Language Models for Base Human Reliability Analysis