[Paper Review] Tutorial: Safe and Reliable Machine Learning

Suchi Saria, Adarsh Subbaswamy|arXiv (Cornell University)|Apr 15, 2019

Adversarial Robustness in Machine Learning17 references40 citations

TL;DR

A tutorial outlining reliability principles for machine learning in high-stakes settings, focusing on failure prevention, failure identification and reliability monitoring, and maintenance, with connections to fairness, transparency, and interpretability.

ABSTRACT

This document serves as a brief overview of the "Safe and Reliable Machine Learning" tutorial given at the 2019 ACM Conference on Fairness, Accountability, and Transparency (FAT* 2019). The talk slides can be found here: https://bit.ly/2Gfsukp, while a video of the talk is available here: https://youtu.be/FGLOCkC4KmE, and a complete list of references for the tutorial here: https://bit.ly/2GdLPme.

Motivation & Objective

• Motivate the need for reliability in ML systems used in high-stakes decision making.
• Summarize core reliability principles and how they relate to fairness, transparency, and interpretability.
• Discuss technical approaches for measuring and ensuring reliability across data, models, and reporting.
• Highlight open problems and future directions in reliable ML deployment.

Proposed method

• Categorize sources of failures into bad data, environment shifts, model errors, and reporting.
• Describe proactive framework for preventing failures via environment shift analysis using DAGs and selection diagrams.
• Discuss robustness approaches such as adversarial training and robustness certificates for model protection.
• Emphasize the role of reporting standards (datasheets, model cards) and reliability documentation.
• Outline failure identification and reliability monitoring methods, including point-wise trust scores and out-of-distribution detection.
• Address maintenance challenges and the notion of technical debt in ML systems.

Experimental results

Research questions

• RQ1 What are the main sources of failures in machine learning systems when deployed in real-world environments?
• RQ2 How can reliability principles be operationalized to prevent, detect, and maintain ML systems against shifts, adversarial inputs, and reporting gaps?
• RQ3 What frameworks (e.g., DAGs, selection diagrams) support proactive invariance against environment shifts?
• RQ4 How should reporting standards incorporate reliability considerations such as robustness certificates and model verification?

Key findings

• Reliability in ML can be structured around three principles: failure prevention, failure identification and reliability monitoring, and maintenance.
• Shifts in environment and dataset bias can cause models to underperform outside training conditions, necessitating proactive generalization strategies.
• Model-related issues include faulty assumptions and fragility to high-dimensional inputs, motivating robust training and verification methods.
• Poor reporting contributes to misuse and misalignment, suggesting the adoption of datasheets, model cards, and reliability-focused documentation.
• Point-wise reliability and trust auditing can help reject unreliable predictions post-training, addressing out-of-distribution and local fit concerns.