[Paper Review] Reliability, Resilience and Human Factors Engineering for Trustworthy AI Systems

Saurabh Mishra, Anand B. Rao+4 more|arXiv (Cornell University)|Nov 13, 2024

Risk and Safety AnalysisDecision Sciences3 citations

TL;DR

This paper proposes an integrated framework combining reliability engineering, resilience engineering, and human factors engineering—centered on Prognostics and Health Management (PHM)—to enhance the trustworthiness of AI systems. By adapting traditional engineering metrics like failure rate and MTBF, and embedding human-centric design principles, the framework enables proactive failure prevention and efficient recovery, demonstrated via real-world system status data from OpenAI, offering a scalable model for AI safety and regulatory alignment.

ABSTRACT

As AI systems become integral to critical operations across industries and services, ensuring their reliability and safety is essential. We offer a framework that integrates established reliability and resilience engineering principles into AI systems. By applying traditional metrics such as failure rate and Mean Time Between Failures (MTBF) along with resilience engineering and human reliability analysis, we propose an integrate framework to manage AI system performance, and prevent or efficiently recover from failures. Our work adapts classical engineering methods to AI systems and outlines a research agenda for future technical studies. We apply our framework to a real-world AI system, using system status data from platforms such as openAI, to demonstrate its practical applicability. This framework aligns with emerging global standards and regulatory frameworks, providing a methodology to enhance the trustworthiness of AI systems. Our aim is to guide policy, regulation, and the development of reliable, safe, and adaptable AI technologies capable of consistent performance in real-world environments.

Motivation & Objective

• Address the growing need for trustworthy AI in critical applications by integrating engineering disciplines traditionally used in complex systems.
• Recognize that technical reliability and human reliability are interdependent and must be addressed together in AI system design.
• Develop a systematic approach to manage AI system performance and recovery from failures using established engineering methodologies.
• Bridge the gap between traditional engineering practices and emerging AI technologies to support policy, regulation, and risk management.
• Provide a practical, data-informed framework for AI developers and regulators to enhance system safety and resilience across the AI lifecycle.

Proposed method

• Apply Prognostics and Health Management (PHM) as a unifying framework to monitor and manage AI system reliability and resilience across lifecycle stages.
• Categorize AI systems into five subsystems—Data, Model, Computing Infrastructure, Code/Software, and Human Interaction—each with distinct failure modes.
• Use reliability engineering techniques such as FMEA (Failure Mode and Effects Analysis) to identify and prioritize failure risks in AI components.
• Integrate human factors engineering throughout the lifecycle, including pre-deployment (design) and post-deployment (response), via the Human-Centric AI Reliability Model (HC-AIRM).
• Apply quantitative metrics like failure rate, MTBF, and probabilistic risk assessment grounded in set theory and sigma algebra to evaluate system performance.
• Demonstrate the framework’s practicality using publicly available system status data from OpenAI, showing real-world applicability to operational AI platforms.

(a) High-Level Overview of AI Failure Modes and Types

Experimental results

Research questions

• RQ1 How can traditional reliability and resilience engineering principles be adapted to address the unique failure modes of AI systems?
• RQ2 What role do human factors play in both preventing and recovering from AI system failures, and how can they be systematically integrated into AI design?
• RQ3 To what extent can established engineering metrics (e.g., MTBF, failure rate) be meaningfully applied to AI systems to ensure trustworthiness?
• RQ4 How can a human-centric PHM framework improve the safety and resilience of AI systems in real-world operational environments?
• RQ5 What are the implications of integrating reliability, resilience, and human factors engineering for AI policy, regulation, and risk management?

Key findings

• The integration of reliability, resilience, and human factors engineering into a unified PHM framework significantly enhances the predictability and trustworthiness of AI systems.
• Failure rates and MTBF metrics—common in aerospace and nuclear engineering—can be meaningfully applied to AI systems to quantify performance and risk.
• Human factors are not peripheral but central to both failure prevention and recovery, with the HC-AIRM model demonstrating their critical role in system design.
• Real-world analysis of OpenAI’s system status data confirms the framework’s practical applicability in monitoring and managing AI system health and performance.
• The framework enables a portfolio-based approach to AI investment, supporting risk-aware decision-making through continuous monitoring and insurance-ready metrics.
• Proactive measures such as watermarking and design-for-fail-safe (e.g., IKEA-style component design) reduce misuse and improve system resilience, especially against adversarial and unintended failures.

(b) Hypothetical 3D Matrix of Sectors (Tasks), Impacts, and Failure Modes