[Paper Review] Towards Practical Verification of Machine Learning: The Case of Computer Vision Systems

Kexin Pei, Zhu, Linjie+4 more|arXiv (Cornell University)|Dec 5, 2017

Adversarial Robustness in Machine Learning35 references71 citations

TL;DR

The paper introduces VeriVis, a blackbox verification framework to test safety properties of computer vision systems under real-world transformations, finding thousands of violations and enabling partial verification and retraining to reduce violations.

ABSTRACT

Due to the increasing usage of machine learning (ML) techniques in security- and safety-critical domains, such as autonomous systems and medical diagnosis, ensuring correct behavior of ML systems, especially for different corner cases, is of growing importance. In this paper, we propose a generic framework for evaluating security and robustness of ML systems using different real-world safety properties. We further design, implement and evaluate VeriVis, a scalable methodology that can verify a diverse set of safety properties for state-of-the-art computer vision systems with only blackbox access. VeriVis leverage different input space reduction techniques for efficient verification of different safety properties. VeriVis is able to find thousands of safety violations in fifteen state-of-the-art computer vision systems including ten Deep Neural Networks (DNNs) such as Inception-v3 and Nvidia's Dave self-driving system with thousands of neurons as well as five commercial third-party vision APIs including Google vision and Clarifai for twelve different safety properties. Furthermore, VeriVis can successfully verify local safety properties, on average, for around 31.7% of the test images. VeriVis finds up to 64.8x more violations than existing gradient-based methods that, unlike VeriVis, cannot ensure non-existence of any violations. Finally, we show that retraining using the safety violations detected by VeriVis can reduce the average number of violations up to 60.2%.

Motivation & Objective

• Motivate the need for robust verification of ML systems in safety-critical domains (e.g., autonomous systems, medical diagnosis).
• Define a generic framework for input-output safety properties under realistic attacker capabilities.
• Develop VeriVis to verify safety properties on state-of-the-art vision systems using blackbox analysis.
• Demonstrate scalability by evaluating across multiple DNNs, commercial APIs, and self-driving car models.

Proposed method

• Propose a formal framework for ML safety properties in input-output space with transformations T(I;c).
• Decompose image transformations into Dependent Pixels (DP) and Dependence Function (DF) to enable space reduction.
• Introduce critical parameter values to reduce the transformation parameter space to a finite set without missing unique outputs.
• Apply a blackbox, dynamic analysis approach to verify properties across 12 transformations and 15 vision systems.
• Show that the number of unique outputs under realistic transforms is polynomial in input size, enabling scalable verification.

Experimental results

Research questions

• RQ1 What real-world safety properties can be verified on computer vision systems using a blackbox approach?
• RQ2 How can input spaces of image transformations be reduced without losing verification guarantees?
• RQ3 To what extent can VeriVis detect safety-property violations across diverse vision models and APIs?
• RQ4 Can retraining on VeriVis-detected violations reduce future violations and improve robustness?

Key findings

• VeriVis finds thousands of safety-property violations across 15 vision systems, including six ImageNet classifiers, five commercial APIs, and four self-driving car models.
• VeriVis identifies up to 64.8x more violations than existing gradient-based methods.
• On average, VeriVis can verify the tested safety properties for about 31.7% of the test images.
• Retraining with Violation-Generated data reduces the average number of violations by up to 60.2%.
• The evaluation encompasses 12 safety properties and 15 systems, demonstrating broad applicability of the approach.