[Paper Review] White Paper Machine Learning in Certified Systems

Hervé Delseny, Christophe Gabreau+26 more|arXiv (Cornell University)|Mar 18, 2021

Adversarial Robustness in Machine Learning118 references17 citations

TL;DR

This white paper investigates the integration of machine learning (ML) into certified systems—particularly in safety-critical domains like aviation—by identifying key challenges related to reliability, verification, and certification. It proposes tailored ML techniques, engineering practices, and certification frameworks to overcome these barriers, demonstrating that ML can be safely embedded in certified systems when supported by rigorous methodologies and tooling.

ABSTRACT

Machine Learning (ML) seems to be one of the most promising solution to automate partially or completely some of the complex tasks currently realized by humans, such as driving vehicles, recognizing voice, etc. It is also an opportunity to implement and embed new capabilities out of the reach of classical implementation techniques. However, ML techniques introduce new potential risks. Therefore, they have only been applied in systems where their benefits are considered worth the increase of risk. In practice, ML techniques raise multiple challenges that could prevent their use in systems submitted to certification constraints. But what are the actual challenges? Can they be overcome by selecting appropriate ML techniques, or by adopting new engineering or certification practices? These are some of the questions addressed by the ML Certification 3 Workgroup (WG) set-up by the Institut de Recherche Technologique Saint Exupéry de Toulouse (IRT), as part of the DEEL Project.

Motivation & Objective

• To identify the core challenges that prevent the adoption of machine learning in certified systems, especially in safety-critical domains such as aviation.
• To assess whether these challenges can be mitigated through appropriate selection of ML techniques, engineering practices, or certification procedures.
• To provide a comprehensive framework and recommendations for integrating ML into systems subject to formal certification processes.
• To support the DEEL Project’s goal of enabling trustworthy and certifiable AI in regulated environments through structured research and collaboration.

Proposed method

• Conduct a systematic analysis of existing ML techniques to evaluate their suitability for safety-critical certification based on transparency, verifiability, and reproducibility.
• Define a set of engineering practices tailored for ML development in certified systems, including data curation, model validation, and testing procedures.
• Propose a certification pathway that aligns ML lifecycle activities with regulatory standards, such as DO-178C for aeronautics.
• Introduce a taxonomy of ML risks and mitigation strategies, focusing on robustness, fairness, and explainability.
• Engage industry and regulatory stakeholders to align technical recommendations with real-world certification requirements.
• Use case studies and expert consensus to validate the proposed framework across different application domains.

Experimental results

Research questions

• RQ1 What are the primary technical and regulatory challenges that hinder the certification of machine learning systems in safety-critical environments?
• RQ2 Which machine learning techniques are most amenable to certification due to their interpretability, determinism, and verifiability?
• RQ3 How can engineering practices be adapted to ensure the reliability and traceability of ML components throughout their lifecycle?
• RQ4 What modifications to existing certification standards are necessary to accommodate machine learning-based systems?
• RQ5 Can a structured framework be established to support the certification of ML components while maintaining safety and compliance?

Key findings

• Many current ML techniques, especially deep learning, pose significant challenges for certification due to their lack of transparency and determinism.
• Simpler, more interpretable models such as decision trees, linear models, and symbolic systems show higher compatibility with certification requirements.
• The integration of ML into certified systems is feasible when supported by rigorous data management, model validation, and testing protocols.
• Certification authorities can adapt existing standards like DO-178C by extending them to include ML-specific development and verification activities.
• A clear separation between training, validation, and deployment phases is essential to ensure repeatability and auditability in certified systems.
• Stakeholder collaboration between industry, regulators, and researchers is critical to establishing a viable certification pathway for ML components.