[Paper Review] Guidance on the Assurance of Machine Learning in Autonomous Systems (AMLAS)

Richard Hawkins, Colin Paterson+4 more|arXiv (Cornell University)|Feb 2, 2021

Adversarial Robustness in Machine Learning37 references47 citations

TL;DR

AMLAS provides a six-stage process and safety-case patterns to systematically assure ML components in autonomous systems, linking system hazards to ML-specific requirements and evidentiary arguments.

ABSTRACT

Machine Learning (ML) is now used in a range of systems with results that are reported to exceed, under certain conditions, human performance. Many of these systems, in domains such as healthcare , automotive and manufacturing, exhibit high degrees of autonomy and are safety critical. Establishing justified confidence in ML forms a core part of the safety case for these systems. In this document we introduce a methodology for the Assurance of Machine Learning for use in Autonomous Systems (AMLAS). AMLAS comprises a set of safety case patterns and a process for (1) systematically integrating safety assurance into the development of ML components and (2) for generating the evidence base for explicitly justifying the acceptable safety of these components when integrated into autonomous system applications.

Motivation & Objective

• Define a stage-wise process to integrate safety assurance into ML component development within autonomous systems.
• Link system-level hazards and risk analyses to ML-specific safety requirements.
• Provide reusable safety-argument patterns to justify ML safety claims with explicit assumptions and uncertainties.
• Enable iterative feedback between ML development and system safety analysis to address interdependencies.
• Produce an explicit safety case for the ML component that can be integrated into the overall system safety case.

Proposed method

• Define six AMLAS stages: ML safety assurance scoping, ML safety requirements, data management, model learning, model verification, and deployment integration.
• Instantiate safety assurance argument patterns using artefacts from preceding activities to justify ML safety claims (GSN-based patterns).
• Translate system safety requirements into ML safety requirements and validate them through reviews and simulation.
• Define data requirements and generate/validate data sets aligned with ML safety requirements.
• Develop and test ML models while documenting learning and data processes with dedicated argument patterns.
• Integrate and test the ML component within the overall system, iterating as necessary based on feedback.

Experimental results

Research questions

• RQ1 How can system-level hazards and safety requirements be effectively mapped to ML-specific safety requirements for autonomous systems?
• RQ2 What artefacts and argument patterns best support a justification that ML components meet safety requirements within an explicit safety case?
• RQ3 How should data management and model learning be documented to enable traceable, auditable assurance for ML components?
• RQ4 What is the role of iterative feedback between ML development stages and system safety analysis in AMLAS?
• RQ5 How can AMLAS be integrated with existing safety standards to govern ML components in safety-critical domains?

Key findings

• AMLAS provides a structured six-stage process to generate an explicit ML safety case for autonomous systems.
• The approach links system safety requirements to ML-specific safety requirements and supports justification via staged safety argument patterns.
• Safety assurance is iterative and allows revisiting stages (e.g., data, learning, verification) based on feedback from verification results.
• AMLAS emphasizes explicit documentation of assumptions, trade-offs, and uncertainties within each stage’s safety argument.
• GSN-based argument patterns are used to document and instantiate safety claims across stages.
• The methodology focuses on off-line supervised learning and acknowledges potential benefits for other ML types like reinforcement learning.