[Paper Review] Trustworthy AI

Jeannette M. Wing|arXiv (Cornell University)|Feb 14, 2020

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper proposes a framework for trustworthy AI by adapting principles from trustworthy computing to address AI's brittleness and unfairness. It advocates formal verification as a core method to ensure reliability, fairness, and robustness, positioning trustworthy AI as an extension of formal methods with new research challenges in verification and validation for AI systems.

ABSTRACT

The promise of AI is huge. AI systems have already achieved good enough performance to be in our streets and in our homes. However, they can be brittle and unfair. For society to reap the benefits of AI systems, society needs to be able to trust them. Inspired by decades of progress in trustworthy computing, we suggest what trustworthy properties would be desired of AI systems. By enumerating a set of new research questions, we explore one approach--formal verification--for ensuring trust in AI. Trustworthy AI ups the ante on both trustworthy computing and formal methods.

Motivation & Objective

• To establish a foundation for trustworthy AI by adapting principles from trustworthy computing to address systemic risks in AI deployment.
• To identify key trustworthy properties—such as reliability, fairness, and robustness—that AI systems must satisfy to gain societal trust.
• To frame formal verification as a central methodology for ensuring that AI systems behave as intended under diverse conditions.
• To define new research challenges at the intersection of formal methods and AI, extending the scope of verification beyond traditional software.
• To position trustworthy AI as a critical advancement that raises the bar for both formal methods and trustworthy computing.

Proposed method

• Adapting the principles of trustworthy computing—such as correctness, availability, and integrity—to the context of AI systems.
• Proposing formal verification as a rigorous technique to mathematically prove desired properties of AI models and systems.
• Integrating formal methods with AI development life cycles to verify behavior under edge cases, adversarial inputs, and distributional shifts.
• Extending formal verification to handle learning-based components, such as neural networks, through abstraction and compositional reasoning.
• Using formal specifications to encode requirements like fairness, robustness, and explainability in machine learning models.
• Establishing a research agenda that bridges formal methods and AI, focusing on scalable verification techniques for complex, data-driven systems.

Experimental results

Research questions

• RQ1 What trustworthy properties—such as fairness, robustness, and reliability—should be prioritized in AI systems to ensure societal trust?
• RQ2 How can formal verification be adapted to verify learning-based AI components, such as deep neural networks?
• RQ3 What new verification techniques are needed to handle the non-determinism and data dependence inherent in modern AI systems?
• RQ4 How can formal methods be integrated into the full AI development lifecycle to ensure end-to-end trustworthiness?
• RQ5 What are the limitations of current formal verification approaches when applied to AI, and how can they be overcome?

Key findings

• Trustworthy AI requires extending formal verification techniques to handle the unique challenges of learning-based systems, such as generalization and distributional shifts.
• Formal verification can provide mathematical guarantees on AI behavior, increasing confidence in safety-critical applications like healthcare and autonomous vehicles.
• The integration of formal methods into AI development introduces new research challenges, particularly in scalability and specification authoring for complex models.
• Trustworthy AI elevates the standards of both trustworthy computing and formal methods, demanding new theoretical and practical advances.
• The paper identifies a critical need for formal specification languages tailored to AI properties like fairness and robustness, which are not easily captured by traditional software specifications.
• The framework positions formal verification not as a post-hoc check but as a foundational component of trustworthy AI system design.