[Paper Review] Architecting Dependable Learning-enabled Autonomous Systems: A Survey

Chih‐Hong Cheng, Dhiraj Gulati+1 more|arXiv (Cornell University)|Feb 27, 2019

Adversarial Robustness in Machine Learning62 references4 citations

TL;DR

This survey proposes a systematic architectural framework for building dependable learning-enabled autonomous systems, focusing on automated driving. It integrates diverse redundancy, information fusion, runtime monitoring, and dependable machine learning techniques—such as uncertainty estimation, interpretable CNNs, and robust optimization—to enhance system reliability, with key contributions in categorizing existing approaches and identifying critical research challenges in verification, adaptive fusion, and interpretable monitoring.

ABSTRACT

We provide a summary over architectural approaches that can be used to construct dependable learning-enabled autonomous systems, with a focus on automated driving. We consider three technology pillars for architecting dependable autonomy, namely diverse redundancy, information fusion, and runtime monitoring. For learning-enabled components, we additionally summarize recent architectural approaches to increase the dependability beyond standard convolutional neural networks. We conclude the study with a list of promising research directions addressing the challenges of existing approaches.

Motivation & Objective

• Address the critical need for dependable architectures in learning-enabled autonomous systems, especially following high-profile autonomous vehicle accidents.
• Systematically categorize architectural approaches for dependability in automated driving, focusing on four pillars: diverse redundancy, information fusion, runtime monitoring, and dependable ML.
• Bridge the gap between existing dependability techniques and their integration into a coherent reference architecture for autonomous systems.
• Highlight challenges in verification, fusion fidelity, and monitor interpretability to guide future research.
• Provide a structured foundation for researchers and practitioners to build safer, more reliable autonomous driving systems.

Proposed method

• Categorize architectural approaches into four key pillars: diverse redundancy, information fusion, runtime monitoring, and dependable machine learning.
• Use diverse redundancy by deploying different algorithms (e.g., stereo vision vs. LiDAR, rule-based vs. ML-based prediction) for the same functional task to avoid common-mode failures.
• Apply information fusion techniques—such as Kalman filters and adaptive fusion—using data from heterogeneous sensors to improve decision accuracy and robustness.
• Implement runtime monitoring to detect anomalies in ML components by measuring deviation from training data distribution or using specification-based checks.
• Integrate dependable ML techniques including Monte Carlo dropout for uncertainty estimation, interpretable CNNs with localized and class-specific neuron activation, and provably robust neural networks via adversarial training.
• Propose feedback loops from monitors to fusion processes to enable adaptive, resilient decision-making under uncertainty.

Experimental results

Research questions

• RQ1 How can diverse redundancy be systematically designed to prevent common-mode failures across different algorithmic implementations in autonomous driving systems?
• RQ2 What are the limitations of current information fusion techniques under non-Gaussian noise and model inaccuracies, and how can fusion be made adaptive to sensor failures?
• RQ3 How can runtime monitors be designed to efficiently detect both known and unknown anomalies in ML components with high detection rates and low computational overhead?
• RQ4 What criteria should guide the prioritization of specifications for runtime monitoring, based on failure occurrence, impact, and controllability?
• RQ5 How can runtime monitoring be extended to runtime enforcement to actively correct or prevent unsafe behaviors in real time?

Key findings

• Diverse redundancy significantly improves system resilience by using different algorithms (e.g., stereo vision, LiDAR, rule-based, or ML-based) for perception, prediction, and planning, reducing the risk of common failure modes.
• Information fusion techniques such as Kalman filters and adaptive fusion improve decision accuracy, but their performance degrades under non-Gaussian noise or rigid fusion architectures, necessitating feedback from runtime monitors.
• Monte Carlo dropout provides a practical method for uncertainty estimation in deep neural networks, with a 90% confidence estimate achievable when 10% of 100 dropout networks deviate.
• Interpretable CNNs enhance model transparency by ensuring neurons activate only for specific object categories and spatial locations, enabling construction of human-readable decision trees.
• Provably robust neural networks trained via adversarial optimization maintain consistent behavior within a bounded neighborhood of training data, improving robustness to input perturbations.
• Runtime monitoring can be enhanced by incorporating interpretable rules (e.g., 'objects cannot be unrealistically large') and by using ML-based techniques to detect unknown unknowns, though this remains an open challenge.